Why your AI agent keeps failing on real websites

Your agent works on the pages you tested it against. Then you point it at the open web and the answers go vague, or confidently wrong, or it says the page was empty when you can see the content in your browser.

The instinct is to reach for the prompt. Usually the prompt is fine. The failure happened before the model saw anything — in the layer that fetches a URL and turns it into text.

We measured that layer on real pages. Here are the five ways it breaks, with what each one actually looks like.

1. The page has no text until JavaScript runs

atlassian.com/software/jira/pricing, fetched the ordinary way, is 1.37 MB of HTML that yields one token of readable content. Not one paragraph — one token, a newline.

The prices are in there. They are inside a <script>, as application state, waiting for a browser to build the page out of them.

Fetch the same URL with a headless browser and it becomes 3,048 tokens that open:

## **Transparent pricing for every team.**

Same URL. Same day. Zero content or the whole page, depending entirely on whether you executed JavaScript.

This is the failure that looks like a model problem and is not. Your agent said "the page does not mention pricing" because, in the bytes it was given, that was true.

2. But rendering is not the fix you want it to be

The obvious response is to render everything. Try it on stripe.com/docs:

FetchError: Rendered fetch failed: Navigation timeout of 25000 ms exceeded

A page that a plain fetch returns in under a second never finished loading in a headless browser inside 25 seconds. The non-rendered version, for all its faults, at least came back.

Rendering costs you a browser process per page, several hundred milliseconds to several seconds of latency, and a new class of failure — timeouts, blocked resources, pages that never fire load. If your agent renders by default, it is slower and less reliable on some of the web, not more.

The workable shape is: cheap fetch first, check whether you got anything, escalate to rendering only for the pages that came back empty. That check is the part people leave out.

3. You are paying for markup you never use

Across twelve well-known pages we measured, a median of under 1% of what the server sends survives to the model. shopify.com/pricing sends 410,515 tokens and yields 2,479.

If your agent stuffs raw HTML into the context window, you are paying for <div> wrappers and inline SVG at input-token prices, and crowding out the pages you also wanted to read. Extract before you send. Better still, ask whether the site will hand you Markdown directly — some already will, and of the twelve pages we tested, five served a Markdown representation of some kind — one of which is ours.

4. Extraction keeps the wrong part

Extraction is a heuristic. It looks for the article and discards the furniture, and on pages that are not shaped like articles it guesses badly.

stripe.com/docs reduces to 1,057 characters that begin:

$ stripe payment_intents create --amount 1099 --currency "usd"

That is a real fragment of the page. It is also a terminal example rather than a sentence describing what Stripe does. An agent given only that has to infer the product from a code sample.

So: never trust that a successful extraction is a useful one. Cheap guard — if what you extracted has no sentence-shaped text, or is shorter than a few hundred characters on a page that sent half a megabyte, treat it as a failed read and escalate, rather than passing it to the model as fact.

5. The same URL does not return the same page

Fetching shopify.com/pricing plainly returned English. Rendering the identical URL from the identical machine returned Spanish — Pago mensual32 € EUR/mes where the plain fetch said Pay monthly€32 EUR/mo.

Nothing is broken here. The headless browser sent different Accept-Language and geolocation signals than our plain fetch did, and got a different, correct representation. But if you cache by URL alone, or compare today's read against yesterday's, that difference will look like the site changed its prices.

Send an explicit Accept-Language. Key your cache on what you actually asked for.

The failure that did not happen

We expected blocking. We probed all twelve pages twice — once as a browser, once identifying as OAI-SearchBot — and compared status codes across Cloudflare, Vercel and CloudFront.

None of them treated the bot differently. Not one.

That does not mean nobody blocks agents; plenty do, and you should still read robots.txt and honour it. It means that if your agent is failing on a broad sample of the web, blocking is unlikely to be the reason. The empty room is far more common than the locked door.

A reading layer that survives contact with the web

1. Ask for Markdown first  → Accept: text/markdown, or /page.md
2. Plain fetch             → cheap, no JS
3. Check what you got      → sentence-shaped? long enough for the bytes sent?
4. Escalate if not         → render, with a timeout and a budget
5. Give up honestly        → "I could not read this page"

Step 5 matters more than it looks. An agent that reports an unreadable page is debuggable. An agent that hands the model a newline and lets it improvise is the one that makes things up about your pricing.

The short version

  • Most agent failures are read failures, not reasoning failures.
  • Empty-without-JS is the single biggest one, and rendering fixes it — at a cost, and not always.
  • Verify every extraction before you trust it.
  • Blocking gets the attention. Emptiness does the damage.

Check how a page reads · What an agent actually sees · Serving Markdown to agents