# What an AI agent actually sees when it visits your site

You open your homepage and see a design: a nav bar, a hero image, three columns of copy, a pricing table, a cookie banner you stopped noticing months ago.

An agent opens the same URL and sees none of that. It gets a stream of bytes, throws most of them away, and hands whatever is left to a model with a fixed context window. What survives that trip is the entire basis on which an assistant will describe your company.

Most people have never looked at it. So we measured.

## The measurement

On 31 August 2026 we ran twelve well-known pages through our own engine: fetch the page the way a plain agent would, extract the readable content, convert it to Markdown, and count tokens at both ends with `cl100k_base`.

| Page | Sent | Reaches the model | Left |
|---|---|---|---|
| `wordpress.org/about` | 56,663 tok | 159 tok | 0.3% |
| `vercel.com/docs` | 278,962 tok | 905 tok | 0.3% |
| `stripe.com/docs` | 376,509 tok | 1,042 tok | 0.3% |
| `cloudflare.com/learning/…` | 95,028 tok | 631 tok | 0.7% |
| `nodejs.org/en/about` | 117,729 tok | 1,190 tok | 1.0% |
| `shopify.com/pricing` | 410,515 tok | 2,479 tok | 0.6% |
| `notion.com/pricing` | 172,136 tok | 2,805 tok | 1.6% |
| `slack.com/pricing` | 294,980 tok | 5,718 tok | 1.9% |
| `developer.mozilla.org/…/HTTP` | 59,350 tok | 3,059 tok | 5.2% |
| `react.dev/learn` | 91,899 tok | 3,804 tok | 4.1% |
| `atlassian.com/software/jira/pricing` | 379,636 tok | 1 tok | 0.0% |
| `agentready.md` | 13,188 tok | 627 tok | 4.8% |

Not one page delivers more than 6% of what it sends. The median is under 1%.

That is not a scandal. Most of those bytes are doing a real job — layout, styling, analytics, interactivity — for the reader with a browser. The point is narrower: **all of it is paid for on the way in, and almost none of it is what the agent came for.**

## The four things that happen to your page

Between the request and the model, your page passes through four gates. Each one drops something.

**1. The fetch.** An agent asks for the URL. Most agent fetchers do not run JavaScript — it is slow, expensive and a security surface. So what it gets is your server's raw response, not the page a browser assembles.

**2. The extraction.** Something has to decide which part of that HTML is the article and which is furniture. This is the same problem Firefox's Reader View solves, and the same library everyone uses for it. Nav, footer, cookie banner and sidebar go in the bin. So does anything the extractor cannot tell apart from a sidebar.

**3. The conversion.** What is left becomes text. Headings, lists, tables, links and code blocks survive as structure. Class names, `<div>` wrappers, inline SVG and `data-` attributes do not.

**4. The context window.** Whatever remains is now competing for space with the user's question, the system prompt, and possibly nine other pages.

Your content only exists, as far as the assistant is concerned, if it makes it through all four.

## What it looks like when it goes well

`vercel.com/docs` reduces 278,962 tokens to 905, and this is how those 905 open:

```markdown
## Ship anything with Vercel

Deploy your app on Vercel in three steps: install the CLI, add agent
support if you need it…
```

That is a clean answer to "what does Vercel do". A model reading it can quote it, summarise it and attribute it. The 99.7% that got dropped cost the agent nothing it wanted.

## What it looks like when it goes wrong

`atlassian.com/software/jira/pricing` sends 1.37 MB and delivers **one token**.

The page is a shell. The HTML contains `<div id="wac-root"></div>` and then about a megabyte of embedded application state — the prices, the plan names, the feature lists, all of it inside a `<script>` as JSON, waiting for JavaScript to build the page. Our extractor dutifully pulled out 1.1 million characters of `window.SSR_detailMetrics=Object.freeze({"getFeatureGateValues":…`, and once scripts were stripped, what remained was a single newline.

So: ask an assistant what Jira costs, and nothing on that page can be the source of its answer. Not because Atlassian blocked anyone — they did not — but because the page has no text in it until a browser runs it.

Fetch the same URL with a headless browser and it yields 3,048 tokens opening with **Transparent pricing for every team**. The content is there. It is just not in the response. [Whether an agent runs a browser, and what that costs it](/blog/why-ai-agents-fail), is the whole difference.

`stripe.com/docs` is a milder version of the same shape: 1.28 MB in, and the extracted content is 1,057 characters that begin with a CLI example rather than a description of Stripe.

## The thing we expected to find and did not

We assumed some of these sites would be turning agents away at the edge. We probed each one twice — once as an ordinary browser, once identifying as `OAI-SearchBot` — and compared.

**Not one of the twelve treated the bot differently.** Same status code both times, across Cloudflare, Vercel and CloudFront. (`cloudflare.com/learning` returned 403 to both of our probes, which is a wall, but not a bot-specific one.)

Blocking is the failure everybody talks about. In this sample it did not happen once. What happened instead was quieter: the door was open and the room was empty.

## Look at your own page

Two commands. What you send:

```bash
curl -s https://yoursite.com/page | wc -c
```

Then read what survives extraction — paste the URL into [the checker](/) and look at the Markdown panel, not the score. The question is not whether the number is big. It is whether the text you are reading is the text you would want an assistant to quote about you.

Most people are surprised twice: by how little there is, and by which parts made it.

## The short version

- An agent sees your raw HTML, usually without running JavaScript, and keeps roughly 1% of it.
- If your content is assembled by JavaScript, that 1% can be nothing at all.
- Being blocked is rare. Being empty is common.
- The fix is not to send less markup. It is to make sure the words are in the response.

[Check your site](/) · [Can AI assistants read your site?](/blog/can-ai-read-your-site) · [Serving Markdown to agents](/blog/markdown-for-agents)