Half Your GPTBot Traffic Is Lying to You

A user-agent header is a string the client sends about itself. Nothing stops anyone from setting it to GPTBot/1.0 or ClaudeBot. Scrapers do this constantly, because sites are less likely to block traffic that looks like a helpful AI crawler. So the second you start counting "AI bot visits" by matching that string, you're counting a pile of imposters alongside the real thing.

This matters more now because AEO reporting leans hard on crawler logs. Teams want to know which engines actually fetch their pages, how often, and which URLs. If a third of your GPTBot line is some data broker in a cheap datacenter wearing OpenAI's name tag, every conclusion you draw downstream is off. You'll credit pages that never got read by a real engine and miss the ones that did.

Start with the obvious: inflated crawl counts make it look like the AI engines love you when they barely visit. That feels good in a deck and it's wrong. Worse, spoofed hits cluster on whatever URLs the scraper wanted, not the pages engines choose, so your "most crawled content" list points you at the wrong work. You optimize for a robot that was never there.

Then there's the diagnostic damage. When you're trying to figure out why ChatGPT won't cite you, the first question is whether it's even reading the page. Fake GPTBot traffic answers that question incorrectly. You see hits, assume access is fine, and go chasing content or schema problems while the real issue might be that the genuine crawler got a timeout or a 403. Bad input data turns a two-day fix into a two-month goose chase.

The fix is not clever, it's just discipline most log tools skip. OpenAI, Anthropic, Perplexity, and Google all publish the IP ranges and verification methods for their crawlers. A real hit should resolve by reverse DNS to the vendor's domain and forward-confirm back to the same IP, and it should originate from a documented range. If a request claims to be GPTBot but comes from an address OpenAI has never announced, it's spoofed. You drop it before it touches a chart.

Crescive runs this check on every crawler hit before anything is counted. We match the source IP against each engine's current published ranges, confirm it with forward-and-reverse DNS, and split your traffic into verified, spoofed, and unverifiable. What you see in Crescive is the crawler activity that actually happened, per engine, per URL. That's the number you can build an AEO strategy on, because it's the only one the imposters didn't touch.

Once the fakes are gone, the picture gets useful fast. You can see that PerplexityBot reads you daily but GPTBot hasn't touched your pricing page in three weeks, and now that gap is a real lead instead of noise. You can watch a verified crawler come back after you ship a fix, which is the before-and-after evidence that turns AEO from a vibe into a measurable line item.

It also changes how you handle access. Blocking or rate-limiting decisions get dangerous when you can't tell a real engine from a scraper, because one wrong rule locks out the crawler you need most. With verification in place you can be aggressive with the imposters and generous with the engines that decide whether AI assistants recommend you. That's the whole point: spend your attention on the bots that can actually move your answers.