How to allow or block AI bots in robots.txt
Most sites decide this once, badly, and never revisit it. Either everything is blocked in a panic about training, taking the customer-facing traffic with it, or nothing is configured at all and the default is whatever a crawler assumes.
There are three sensible positions. Here is the file for each, and the mistakes that undo them.
First, the distinction that decides everything
AI companies run two kinds of agent, and they are not interchangeable.
Crawlers collect pages in bulk, on their own schedule, usually for training or indexing: GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended, Applebot-Extended.
On-demand fetchers retrieve one page because a person just asked a question about it: ChatGPT-User, Claude-User, Perplexity-User. Nobody is training on that request. Somebody is waiting for an answer about you.
Blocking the first group is a business decision about your content. Blocking the second is closer to blocking a visitor who typed your URL. Most of the "block all AI" advice you will read makes no distinction between them.
Case 1 — Let them in
The right default for anything that wants to be found: documentation, publishers who want attribution, businesses that want to be recommended.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?session=
Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://example.com/sitemap.xml
Note what that says: come in, index me, use me to answer questions, do not train on me. That is a position robots.txt alone cannot express — see the Content-Signal section below.
Case 2 — Answers yes, training no
The one most businesses actually want, and the one almost nobody configures correctly.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://example.com/sitemap.xml
Two things worth knowing about that file. Google-Extended governs Gemini training and not Google Search — blocking it does not affect your search ranking. And Applebot-Extended works the same way for Apple Intelligence, separately from the Applebot that powers Siri and Spotlight.
Case 3 — Block AI entirely
Legitimate for paywalled archives, licensed material and anything you sell access to.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
Disallow: /
User-agent: *
Allow: /
Content-Signal: search=no, ai-input=no, ai-train=no
Sitemap: https://example.com/sitemap.xml
Stacked User-agent lines share the rules that follow them, which is valid and far easier to maintain than fourteen separate blocks.
Understand the cost before choosing this: you disappear from AI answers. When somebody asks an assistant about your industry, your competitors get named and you do not.
The mistakes
Only the most specific group applies. A crawler matching its own name ignores User-agent: * completely. So this does nothing:
User-agent: *
Disallow: /private/
User-agent: GPTBot
Disallow: /
GPTBot is blocked everywhere, but every other bot reads only the * group — and if you later add an Allow under GPTBot, /private/ stops being protected from it, because that group never mentioned /private/.
Blank lines separate groups. A blank line in the middle of a group ends it. The lines after it belong to nobody until the next User-agent.
Disallow: with nothing after it means allow everything. Disallow: / blocks everything. One character between an open door and a closed one.
robots.txt is not access control. It is a request that well-behaved crawlers honour. Scrapers that ignore it will keep ignoring it. If something must not be read, it needs authentication, not a directive.
Crawl-delay is widely ignored by AI crawlers. If load is the problem, rate-limit at the edge.
Content-Signal: saying what, not just whether
robots.txt answers one question: may you fetch this. It has no way to say "index me but do not train on me", which is the position most sites actually hold.
Content-Signal adds that layer with three independent signals:
Content-Signal: search=yes, ai-input=yes, ai-train=no
search— may this appear in a search indexai-input— may this be used as material for an answer, with attributionai-train— may this be used to train a model
It goes in robots.txt as above, and can also travel as an HTTP header on every response, which is what we do. It is a young convention rather than a standard anyone is obliged to honour — but it costs a line, it is machine-readable, and it states an intent that a Disallow cannot.
Check that yours arrives
Writing the file is not the same as serving it:
curl -s https://example.com/robots.txt
curl -sI https://example.com/robots.txt | grep -i "server\|cf-cache-status"
The second command matters more than it looks. If you are behind a CDN without a robots.txt of your own, the CDN may answer for you with a file you never wrote — a plausible 200 that contains none of your rules and no sitemap. Ours did exactly that for months while our origin returned 404.
Generate your robots.txt · Check what you serve · Are the bots allowed in?