# Content-Signal: telling AI what it may do with your content
`robots.txt` answers one question and has only ever answered one: may you fetch this. Yes or no, path by path, bot by bot. It has no vocabulary for the position most site owners actually hold — index me, quote me, send me readers, do not train a model on my writing.
So people reach for the only lever they have. They `Disallow` everything, and lose the traffic along with the training.
## Fetch permission is not use permission
The two questions are genuinely separate, and only one of them is about your bandwidth. A crawler that reads your pricing page and cites it when somebody asks what to buy is doing you a favour. The same bytes folded into a training set is a different transaction entirely, and one you are not part of.
Blocking everything collapses both into a single `no`. For a paywalled archive that may be right. For a business that wants to be found and recommended, it is the wrong default — you disappear from the answers, and your competitors do not.
Content-Signal is the layer that lets you split them.
## The three signals
```
Content-Signal: search=yes, ai-input=yes, ai-train=no
```
- **`search`** — building a search index and returning results: links and short excerpts. Not AI-generated summaries; those are the next one.
- **`ai-input`** — feeding your page into a model at answer time. Retrieval-augmented generation, grounding, the sources listed under a generated answer.
- **`ai-train`** — training or fine-tuning a model on your content.
They are independent. Any combination is valid, and several of them are sensible positions rather than mistakes.
Values are `yes` and `no`. There is no `maybe`, no per-path syntax, no expiry date.
Leaving a signal out is not a `no`. Under the convention, a use you say nothing about is neither granted nor restricted — you simply have not answered. Our generator emits all three every time, which is the right habit: silence is the one reply nobody can act on.
## Where it goes
Two places, and both are cheap.
In robots.txt, inside a user-agent group:
```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://example.com/sitemap.xml
```
And as an HTTP response header:
```
Content-Signal: search=yes, ai-input=yes, ai-train=no
```
agentready.md serves both. That exact header goes out on every response — every page, every `.md` version, every API reply.
For nginx:
```
add_header Content-Signal "search=yes, ai-input=yes, ai-train=no" always;
```
For Apache:
```
Header set Content-Signal "search=yes, ai-input=yes, ai-train=no"
```
The header earns its place. robots.txt is fetched from your root, occasionally, and an agent that lands on one deep URL because a person pasted a link may never read it at all. The header travels attached to the thing being taken.
There is a meta tag form as well, ``. Use it when the server config is not yours to edit. A header is better, because it also covers responses that are not HTML.
## Three real positions
**A publisher who wants attribution.**
```
Content-Signal: search=yes, ai-input=yes, ai-train=no
```
Appearing inside an answer with a link is distribution — the modern equivalent of being cited. Training is the part nobody is paying for, and the part that does not send anyone back.
**A SaaS that wants to be recommended.**
```
Content-Signal: search=yes, ai-input=yes, ai-train=yes
```
Docs, pricing, changelog. When somebody asks an assistant what to use for a job, you want to be the answer, and being in the training data is how you get named when the model fetches nothing at all. There is nothing here to protect. The material is marketing.
**A paywalled archive.**
```
Content-Signal: search=yes, ai-input=no, ai-train=no
```
You want to be findable, because the search result is the sales pitch. You do not want the substance of the article arriving free inside somebody's answer. This is the case where the signal does work a `Disallow` cannot: blocking the crawl would take your search listing with it.
If one bot deserves different terms, it gets its own group:
```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
```
Normal robots.txt precedence applies: a bot that finds its own name reads that group and ignores `*` entirely.
## What it actually is
A young convention. Not an RFC, not a W3C standard, not something any crawler is obliged to honour. Nobody has promised compliance in a form you could hold them to. If you read that Content-Signal protects your content, you are reading marketing.
So why write the line.
It is machine-readable and it costs almost nothing — one line in a file you already serve, one header you already have a config block for. The preamble that ships with the convention frames a `no` as an express reservation of rights under Article 4 of EU Directive 2019/790, the text-and-data-mining opt-out. Whether that holds in your jurisdiction is a question for a lawyer. An unstated preference is worth nothing anywhere.
And it says a thing `Disallow` has no words for. "Do not train on me" and "go away" are different sentences, and until now you could only send the second one.
## Check that yours arrives
```bash
curl -sI https://example.com/ | grep -i content-signal
curl -s https://example.com/robots.txt | grep -i content-signal
```
Then repeat the first one against a page that is not the homepage. If the header is set inside a `location` block, it may cover less of the site than you think.
One more, if you sit behind a CDN:
```bash
curl -sI https://example.com/robots.txt | grep -i "server\|cf-cache-status"
```
Confirm the robots.txt coming back is yours. Ours returned `404` from origin for months while Cloudflare answered at the edge with a managed policy file we had never written — every check we ran got a plausible `200`.
[Generate your Content-Signal](/tools/content-signal-generator) · [Generate your robots.txt](/tools/robots-txt-generator)