The page looks done because you're looking at it wrong
Open the product page. The price is there. Stock says in stock. The spec table is populated, the reviews load, everything looks shipped. Your QA checklist passes because a human with a browser is doing the checking, and a browser executes JavaScript.
AI crawlers mostly don't. GPTBot, ClaudeBot, PerplexityBot and the rest fetch your raw HTML and read what's in it. They don't spin up a full browser, wait for scripts to run, hydrate the DOM, and call your pricing API. That's expensive at web scale, so they skip it. If your $49 price gets injected by a script after the page loads, the crawler's copy of that page has an empty slot where the number should be.
So when someone asks an assistant what your product costs or whether it's available, the model is working from a version of your page that never had those facts. It either says it doesn't know, guesses, or pulls a stale number from somewhere else. You look either incomplete or wrong, and you never see why, because on your screen the page is perfect.
Same URL, two very different documents
| Comparison category | Field | Human in a browser | AI crawler reading raw HTML |
|---|---|---|---|
| Product title | Visible | Visible | |
| Price | $49/mo | (empty — JS not run) | |
| Availability | In stock | (empty — JS not run) | |
| Spec comparison table | Fully populated | Skeleton placeholder only | |
| Reviews & rating | 4.7 from 320 reviews | (empty — loaded via API) |
Why a normal SEO check misses this
Google's crawler does render JavaScript, at least on a delay, so a lot of teams assume everyone does. That assumption is the trap. The moment you optimize for how AI assistants describe you, you're dealing with a different set of bots that behave more like curl than like Chrome.
This is the specific gap Crescive is built to catch. It fetches your pages the way these crawlers do, parsing the raw HTML instead of a rendered browser session, and then diffs that against what a human sees. It flags the exact fields that exist for people but not for machines: the price that's JS-only, the availability that comes from an API call, the spec row that's a placeholder until hydration. You get a field-by-field list instead of a vague 'your JavaScript might be a problem' warning.
From there the fix is an engineering decision, not a guessing game. You know precisely which values need to live in the initial HTML, and Crescive re-checks after the change to confirm the crawler now sees them. Then it tracks whether the assistants start quoting the right numbers.
How to check whether your key facts survive without JavaScript
- Pick a page where a specific fact matters most: a price, a plan tier, stock status, a spec.
- In your browser, open the page source with View Source (Ctrl+U), not Inspect Element. View Source shows raw HTML; Inspect shows the rendered DOM after scripts run.
- Use Find (Ctrl+F) to search that raw source for the exact value, like '49' or 'In stock'. If it's not there, an AI crawler probably can't see it either.
- Confirm by fetching the page with JavaScript disabled, or run curl on the URL and grep for the value. Same result means the same blind spot.
- For every fact that's missing, decide the fix: server-side render it, pre-render at build time, or inject it into the initial HTML response.
- Re-fetch the raw HTML after deploying and confirm the value is now present before any script runs.
- Repeat for your highest-intent pages, since that's where a wrong or missing answer costs you the most.
The fix is boring, which is good
You don't need to rebuild the site. You need the facts that drive buying decisions to be in the HTML the server sends, before any script executes. Server-side rendering, static pre-rendering, or even a small block of markup with the current price and stock status will do it. The rendering can stay fancy for humans as long as the raw document isn't empty for machines.
Prioritize by intent. Marketing copy that's JS-rendered is a minor issue. A price, a plan comparison, or an availability flag that's JS-only is the thing that makes an assistant misquote you to a ready-to-buy customer. Fix those first, verify in raw HTML, and move down the list.
Key takeaways
- Most AI crawlers read raw HTML and skip client-side JavaScript, so any fact rendered by a script after load is effectively blank to them.
- Your QA passes because a browser runs the JS you're testing with; the crawler doesn't, which is why the gap stays invisible.
- Check with View Source or curl, not Inspect Element, and make sure prices, specs, and availability live in the initial HTML response.
FAQ
Do AI crawlers like GPTBot and ClaudeBot execute JavaScript?
Generally no. Crawlers such as GPTBot, ClaudeBot, and PerplexityBot fetch and parse raw HTML rather than running a full browser session that executes client-side JavaScript. Any content injected by a script after the page loads — a price, stock status, or spec table — is often absent from the version these crawlers actually read, even though it appears normal to a human visitor.
How do I tell if my pricing is invisible to AI crawlers?
Open the page and use View Source (Ctrl+U) to see the raw HTML, not Inspect Element, which shows the rendered DOM after scripts run. Search that source for the exact price. If the number isn't there, an AI crawler likely can't see it. You can confirm by loading the page with JavaScript disabled or fetching it with curl. Crescive automates this by crawling pages the way these bots do and flagging each field that exists for humans but not for machines.