Google Open Sources its Robots.txt parser code

As a part of its efforts to standardize the robotic exclusion protocol, Google introduced yesterday that it was sourcing open supply with the assistance of its robots.txt parser. Which means that the way in which GoogleBot reads and listens for robots.txt information will probably be out there for any robotic or coder to look at or use.

It's uncommon for Google to share what it's doing in primary analysis with the world of open supply – that's its secret sauce – however right here Google has printed it so that every one can entry it.

Google wrote that they "open supply" the C ++ library that our manufacturing methods use to parse and match the foundations in robots.txt information. This library has existed for 20 years and accommodates items of code written within the 90s. Since then, the library has developed; we've discovered loads about how site owners write robots.txt information and the circumstances we have to cowl, and we've added what we've discovered through the years to the draft Web when it is sensible. "

Dialogue Discussion board at Twitter.