Two lines in a robots.txt file decide whether an AI assistant can cite you. They look almost identical and do opposite things.
The three kinds of crawler
Search crawlers. Googlebot, Bingbot. Block them and your pages leave search. Nobody does this on purpose.
Answer crawlers. OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot. These fetch pages when an assistant searches the web to answer a question, and what they fetch is what gets cited, with a link. Block them and the assistant cannot use your page, no matter how good it is. It will use a competitor's instead.
Training crawlers. GPTBot, ClaudeBot, CCBot, and Google-Extended. These collect text to train models. Blocking them is a defensible business decision, and it does not affect whether an assistant can cite you today.
The confusion costs real citations. Plenty of sites blocked "the OpenAI bot" in 2023, meaning GPTBot, and blocked OAI-SearchBot alongside it. They gave up nothing to model training that they were going to be paid for, and gave up being quoted in ChatGPT's answers.
Google-Extended deserves its own line: it controls use in Gemini, and Google documents that it has no effect on Google Search. Blocking it does not remove you from search results.
A robots.txt that reflects that
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
That is a site that wants to be cited and does not want to be training data. The opposite choice is just as valid; what is not valid is making it by accident.
Two things that catch people out
Blocking a page in robots.txt does not remove it from Google. A blocked URL can still be indexed from links, without its content, which is the worst of both. To keep a page out, allow crawling and use a noindex tag, so the crawler can actually see the instruction.
A server error on robots.txt is not "no rules". A 404 means everything is allowed. A 500 means Google treats the whole site as disallowed while the error lasts.
Check it rather than guess
Our robots.txt tester takes any URL, applies the rules the way Google does (longest match wins, and an Allow beats an equal-length Disallow), and tells you crawler by crawler whether the page can be read, and which line decided. It groups them by what they are for, so a blocked training crawler shows as a note and a blocked answer crawler shows as a warning.
Being allowed is only the first condition for being cited. The rest is whether your page answers the question better than the one the assistant found instead, which is what the monthly AI-answer check measures.