Google: Messages on blocked sources, though the robots.txt file has not been modified – a doable motive


If messages about blocked sources seem within the Google Search Console, it might be as a result of the robots.txt file was quickly unavailable.

Google at all times calls robots.txt earlier than crawling an internet site, offered that this file is saved. If messages about blocked sources seem within the Google Search Console, though a robots.txt file exists and has not been modified, this can be resulting from the truth that Google couldn’t quickly entry the robots.txt file.

If a 403 or 500 standing is distributed when retrieving robots.txt, Google won’t crawl the web site in query. That explains Johannes Müller in a current webmaster bubble, Such messages could then seem.

To check whether or not momentary unavailability is the explanation for messages on blocked sources, Müller recommends consulting the log information. There you have to verify the robots.txt calls and see what server standing and response occasions have been related to the calls.

If the robots.txt file is now not accessible and, for instance, a Standing 500 is being despatched completely, Google will cease the crawling for a sure time. That is problematic when new content material is added to an internet site or current content material is modified. After some time it might beGoogle will resume crawling even when the robots.txt file just isn’t obtainable.

Cowl picture: Copyright Patrick Daxenbichler –

Christian Kunz

By Christian Kunz

search engine optimization knowledgeable. Do you want recommendation on your web site? Click on right here.

Clixado show

Article publications on stable magazines and blogs

We cooperate with numerous publishers and bloggers and might due to this fact supply article spots on virtually 4000 blogs on virtually all topics:

    – Creation of lasting hyperlinks, no search engine optimization community
    – Excessive visibility values, no expired domains
    – Single cost, no contract

For every article publication, we create prime quality content material with at the least 400 phrases and publish the article with a DoFollow Bach hyperlink to your web page in {a magazine} or weblog of your selection.

Ask us for examples with out obligation