Google Announces Open Sourcing Of Its Robots.txt Parser
Google has open sourced the C++ library that it has been using for parsing and matching rules in robots.txt files. According to the company, this library has been in use for 20 years and it contains codes that were written in the 90's.
"Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense." wrote the company
Google is also including a testing tool in the open source package to enable developers to test a few rules.
You can visit the GitHub repository for the robots.txt parser here.