After 25 years: Google needs to standardize the robots.txt file and provides its personal robots.txt analyzer to obtain.


The Web has modified dramatically over the previous 20 years and continues to be prolonged to new applied sciences and new alternatives. A really fundamental a part of the robotic's infrastructure, nonetheless, has not modified perpetually, however remains to be not standardized: the robots.txt file. Now Google needs to push normalization and has additionally launched a robots.txt parser.

Nearly all web sites have a robots.txt file that’s neither related nor fascinating to customers or guests of the web site, however ought to be taken into consideration by search engines like google or their robots. There isn’t any obligation to notice, however it’s a good sound and can be applied by all recognized search engines like google. With the file, that is the primary use case. Some information and folders of particular search engine robots could also be excluded.

<img src = "” alt=”googlebot robots "width =" 1500 "peak =" 750 "class =" full-size alignnone wp-image-88798 "srcset =" 1500w,×150.jpg 300w,×384.jpg 768w,×512.jpg 1024w,×320.jpg 640w,×400.jpg 800w "values ​​=" (most width: 1500px) 100vw, 1500px”/>

Who would have thought: the format robots.txt file had already been launched in 1994, however remains to be not standardized. Because of this, there are some variations in implementation and misunderstandings can happen with crawlers. Google now needs to vary with two new initiatives: First, format standardization and implementation have to be realized, with some new options comparable to caching, character set, or advisable dimension to be decided.

So as to have the ability to globally implement these attainable modifications, the evaluation used internally is now out there in open supply and provides this performance for each obtain and take a look at software. There isn’t any clear net service with an API, which is a bit stunning on this case. Google factors out that elements of the analyzer written in C ++ date again to the 1990s, however nonetheless work nicely and are nonetheless used internally right this moment.

»Google robots.txt analyzer
"Announcement of the analyzer
»Standardization Announcement

" Site owners, beware: Google doesn’t take note of all the foundations of robots.txt – these are the alternate options

See as nicely
" Research: Google responds each second of the question and fewer and fewer clicks on search outcomes

" Googlebot turns into everlasting: Chrome's engine will probably be up to date usually

Cease Lacking Google Information: Subscribe to the Google Watchatch E-newsletter
Subscribe to the GoogleWatchBlog publication