# How to allow or block AI bots in robots.txt Most sites decide this once, badly, and never revisit it. Either everything is blocked in a panic about training, taking the customer-facing traffic with it, or nothing is configured at all and the default is whatever a crawler assumes. There are three sensible positions. Here is the file for each, and the mistakes that undo them. ## First, the distinction that decides everything AI companies run two kinds of agent, and they are not interchangeable. **Crawlers** collect pages in bulk, on their own schedule, usually for training or indexing: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `CCBot`, `Bytespider`, `Google-Extended`, `Applebot-Extended`. **On-demand fetchers** retrieve one page because a person just asked a question about it: `ChatGPT-User`, `Claude-User`, `Perplexity-User`. Nobody is training on that request. Somebody is waiting for an answer about you. Blocking the first group is a business decision about your content. Blocking the second is closer to blocking a visitor who typed your URL. Most of the "block all AI" advice you will read makes no distinction between them. ## Case 1 — Let them in The right default for anything that wants to be found: documentation, publishers who want attribution, businesses that want to be recommended. ``` User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /*?session= Content-Signal: search=yes, ai-input=yes, ai-train=no Sitemap: https://example.com/sitemap.xml ``` Note what that says: come in, index me, use me to answer questions, do not train on me. That is a position robots.txt alone cannot express — see the Content-Signal section below. ## Case 2 — Answers yes, training no The one most businesses actually want, and the one almost nobody configures correctly. ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / User-agent: * Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=no Sitemap: https://example.com/sitemap.xml ``` Two things worth knowing about that file. `Google-Extended` governs Gemini training and **not** Google Search — blocking it does not affect your search ranking. And `Applebot-Extended` works the same way for Apple Intelligence, separately from the `Applebot` that powers Siri and Spotlight. ## Case 3 — Block AI entirely Legitimate for paywalled archives, licensed material and anything you sell access to. ``` User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: CCBot User-agent: Bytespider User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent Disallow: / User-agent: * Allow: / Content-Signal: search=no, ai-input=no, ai-train=no Sitemap: https://example.com/sitemap.xml ``` Stacked `User-agent` lines share the rules that follow them, which is valid and far easier to maintain than fourteen separate blocks. Understand the cost before choosing this: you disappear from AI answers. When somebody asks an assistant about your industry, your competitors get named and you do not. ## The mistakes **Only the most specific group applies.** A crawler matching its own name ignores `User-agent: *` completely. So this does nothing: ``` User-agent: * Disallow: /private/ User-agent: GPTBot Disallow: / ``` `GPTBot` is blocked everywhere, but every other bot reads only the `*` group — and if you later add an `Allow` under `GPTBot`, `/private/` stops being protected from it, because that group never mentioned `/private/`. **Blank lines separate groups.** A blank line in the middle of a group ends it. The lines after it belong to nobody until the next `User-agent`. **`Disallow:` with nothing after it means allow everything.** `Disallow: /` blocks everything. One character between an open door and a closed one. **robots.txt is not access control.** It is a request that well-behaved crawlers honour. Scrapers that ignore it will keep ignoring it. If something must not be read, it needs authentication, not a directive. **Crawl-delay is widely ignored** by AI crawlers. If load is the problem, rate-limit at the edge. ## Content-Signal: saying what, not just whether robots.txt answers one question: may you fetch this. It has no way to say "index me but do not train on me", which is the position most sites actually hold. `Content-Signal` adds that layer with three independent signals: ``` Content-Signal: search=yes, ai-input=yes, ai-train=no ``` - `search` — may this appear in a search index - `ai-input` — may this be used as material for an answer, with attribution - `ai-train` — may this be used to train a model It goes in robots.txt as above, and can also travel as an HTTP header on every response, which is what we do. It is a young convention rather than a standard anyone is obliged to honour — but it costs a line, it is machine-readable, and it states an intent that a `Disallow` cannot. ## Check that yours arrives Writing the file is not the same as serving it: ```bash curl -s https://example.com/robots.txt curl -sI https://example.com/robots.txt | grep -i "server\|cf-cache-status" ``` The second command matters more than it looks. If you are behind a CDN without a robots.txt of your own, the CDN may answer for you with a file you never wrote — a plausible `200` that contains none of your rules and no sitemap. Ours did exactly that for months while our origin returned `404`. [Generate your robots.txt](/tools/robots-txt-generator) · [Check what you serve](/tools/robots-txt-checker) · [Are the bots allowed in?](/tools/ai-crawlers-checker)