Skip to content
AEOptimiz

Free, no sign-up

Robots.txt tester

Enter any page. We read the site's robots.txt the way Google does and tell you, crawler by crawler, whether the page may be read and which line decides. AI assistants cannot cite a page their crawler was told to stay out of.

How to use the robots.txt tester

Three steps. No account, and it works on any public page.

  1. 1

    Enter any page

    We find the site's robots.txt from the address.

  2. 2

    We apply the rules like Google

    The most specific matching rule wins, and Allow wins a tie, as in RFC 9309 and Google's documentation.

  3. 3

    Read it crawler by crawler

    Search engines and AI assistants each get a verdict and the exact line that decided it.

What the results mean

  • Allowed

    The crawler may fetch the page. No rule matched, or an Allow rule won.

  • Blocked

    A Disallow rule matches the address for that crawler, either in its own group or in the * group it falls back to.

  • No robots.txt

    A 404 means every crawler may fetch everything. A server error is different: Google treats it as a temporary block on the whole site.

Search crawlers, AI crawlers and training crawlers

Blocking Googlebot or Bingbot keeps pages out of search. Blocking the crawlers AI assistants search with, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, means those assistants cannot read the page to cite it.

Training crawlers are separate. GPTBot, ClaudeBot and CCBot collect text for training models, and Google-Extended controls use in Gemini without affecting Google Search. You can block training and still allow the crawlers that cite you.

Questions

Does blocking a page in robots.txt remove it from Google?

No. A blocked page can still be indexed from links, without its content. To keep a page out, allow crawling and add noindex, so Google can see the noindex.

Which rule wins when two match?

The one with the longest path. If an Allow and a Disallow are the same length, Allow wins.

Do all AI companies respect robots.txt?

Their documented crawlers do. Some owners say a fetch a person asks for directly, such as ChatGPT-User opening a link in a chat, may not. This tool shows what the file asks, not what any crawler does.

This is one check. AEOptimiz runs all of them, every night.

Connect Search Console and it asks Google which of your pages are indexed, why the rest are not, and which fixes are worth the most clicks. Then it checks whether AI assistants cite you.

Run a free coverage report