# How to check whether ChatGPT, Claude and Perplexity can read your site

When someone asks an assistant about your company, one of two things happens. Either it already knows something from training data, months out of date, or it fetches your site right then and reads it.

The second case is the one you can influence, and three things decide how it goes.

## 1. Does your robots.txt let them in?

AI companies use separate user-agents for separate jobs, and blocking one does not block the others. These are the ones that matter today:

| User-agent | Company | What it does |
|---|---|---|
| `GPTBot` | OpenAI | Crawls for training |
| `OAI-SearchBot` | OpenAI | Indexes for ChatGPT search |
| `ChatGPT-User` | OpenAI | Fetches a page because a user asked right now |
| `ClaudeBot` | Anthropic | Crawls for training |
| `Claude-SearchBot` | Anthropic | Indexes for search |
| `Claude-User` | Anthropic | Fetches on behalf of a user |
| `PerplexityBot` | Perplexity | Indexes for answers |
| `Perplexity-User` | Perplexity | Fetches on request |
| `Google-Extended` | Google | Gates Gemini training, not Search |
| `Applebot-Extended` | Apple | Gates Apple Intelligence training |
| `CCBot` | Common Crawl | Feeds many other datasets |
| `Bytespider` | ByteDance | Crawls for training |

The distinction that catches people out: **the "-User" agents are not crawlers**. They fetch because a person asked a question about your page, right that second. Blocking `GPTBot` stops training. Blocking `ChatGPT-User` stops a potential customer from getting an answer about you. Most sites want the first without the second.

Check what you actually serve:

```bash
curl -s https://yoursite.com/robots.txt
```

Read it as a bot would: `Disallow: /` under `User-agent: *` blocks everyone who has no rule of their own, and that includes every agent above.

**One trap worth knowing.** If you are behind Cloudflare and have no robots.txt of your own, Cloudflare may serve one for you — its managed Content Signals Policy. You will get a `200` and a plausible-looking file that you did not write, containing none of your rules and none of your sitemap. We had exactly this for months: our origin returned `404` for `/robots.txt` and nobody noticed, because every check we ran got a `200` from the edge. To tell them apart, ask your origin directly:

```bash
curl -sI https://yoursite.com/robots.txt | grep -i "server\|cf-cache-status"
```

## 2. Does your content survive without JavaScript?

Most AI crawlers do not run JavaScript. If your page renders client-side, what they receive is an empty shell.

```bash
curl -s https://yoursite.com/your-page | wc -c
curl -s https://yoursite.com/your-page | grep -o "<p>" | wc -l
```

A few hundred bytes and no paragraphs means the page is empty as far as a crawler is concerned, however good it looks in a browser. Server-side rendering, static generation or prerendering all fix it; which one hardly matters.

## 3. How much is left after extraction?

Even with the page fetched and rendered, an agent throws away navigation, footers, cookie banners and most markup, then keeps what looks like content. That remainder is what gets quoted about you.

This is the part nobody checks, and it is usually where the surprise is. The fastest way to see it is to read your page as an agent would keep it — stripped to plain text, with the scaffolding gone. If what is left is two sentences and a list of menu items, no amount of robots.txt tuning will help.

## Also worth having

**`llms.txt`** at your root: a short Markdown map of your important pages. It is a young convention rather than a standard anybody has to honour, but it costs almost nothing and it is read.

**`Content-Signal`**, which lets you say what may be done with your content — indexed for search, used as input to an answer, used for training — rather than only whether it can be fetched. Blocking everything is not the only option, and for most businesses it is the wrong one.

**Clean semantic HTML.** Real headings, `<article>`, `<main>`, alt text. It is the same work that helps screen readers, and it is what makes extraction keep the right parts.

## The short version

Run these three, in this order:

1. `curl -s https://yoursite.com/robots.txt` — can they in?
2. `curl -s https://yoursite.com/page | wc -c` — is there anything there without JavaScript?
3. Read what survives extraction — is it what you would want quoted?

Most sites pass the first, fail the third, and never find out.

[Check your site](/) · [robots.txt for AI bots](/tools/robots-txt-generator) · [Are the bots allowed in?](/tools/ai-crawlers-checker)