One line of defense, a whole channel gone
Here's a story that plays out more often than anyone admits. Two years ago a scraper was hammering your site, so someone on the team dropped a broad block into robots.txt to shut it down. It worked. The scraper went away, the pager stopped, and nobody touched the file again.
But robots.txt is not a museum piece. New crawlers launch constantly, and a lot of them are the AI answer engines your future customers are now asking for recommendations. GPTBot showed up in 2023. PerplexityBot and ClaudeBot arrived with names nobody on your team had a reason to recognize. If your old rule was a wildcard, or a wide path disallow, those new bots walked up to your door and got the same 'go away' the scraper did.
The cruel part is that nothing breaks visibly. Google still crawls you, rankings hold, analytics look normal. You are fully indexed and functionally invisible at the same time. The engine that decides whether ChatGPT mentions your brand never got to read a single page, and no dashboard you check daily will tell you that.
Why this hides so well
Robots.txt rules get added reactively. Someone is solving a specific problem in a specific hour, and the fix is scoped to that moment. The trouble is that the file uses user-agent matching, and a User-agent: * block with a broad Disallow catches every crawler that doesn't have its own named exception. So a rule aimed at one bad actor becomes a blanket policy against a category that didn't exist yet.
Then the new bots arrive under unfamiliar names. Your team sees GPTBot in the logs and has no reason to know it's the crawler feeding ChatGPT's answers. It reads like traffic, not opportunity. Nobody maps 'this user-agent string' to 'this is an answer engine that could be citing us,' so the block never gets questioned.
This is exactly the gap Crescive's crawler analytics is built to catch. It watches which AI bots actually reach your pages, cross-references them against your current robots.txt, and flags the ones your rules are turning away by name. Instead of guessing whether PerplexityBot can read you, you see a plain line: blocked, and blocked by which rule. A stale exclusion stops being a silent tax on your citations because someone finally sees it.
Audit and fix your AI crawler access in one sitting
- Pull your live robots.txt at yourdomain.com/robots.txt and read every User-agent and Disallow line out loud. If you see User-agent: * followed by Disallow: / or a broad path, assume it's catching AI bots too.
- List the answer-engine crawlers you care about: GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot and Claude-Web (Anthropic), Google-Extended (Gemini/AI Overviews training), and CCBot if you're comfortable with Common Crawl.
- Check your server logs or CDN analytics for those user-agent strings over the last 30 days. Zero hits from a bot that should be visiting is a red flag it's being blocked or never allowed.
- Decide your policy deliberately. If you want to appear in AI answers, add explicit Allow rules for each named bot above your catch-all block so they aren't swept up by the wildcard.
- Keep the original defensive rule only if it still targets the actual bad actor. Narrow it to that user-agent instead of leaving a wildcard that catches everything.
- Test the new file in Google's robots.txt tester and by fetching a page as each user-agent, then redeploy and re-check logs a week later to confirm the AI bots are now getting through.
Make the file a decision, not an accident
The point isn't that you must open the gates to every crawler. Some brands have real reasons to block AI training bots. The point is that the choice should be made on purpose, this quarter, with the current list of engines in front of you, not inherited from a panic fix you made when the crawler landscape looked completely different.
Set a recurring reminder to re-read robots.txt every quarter, because the roster of AI crawlers keeps growing and your two-year-old rule has no way to keep up on its own. Treat the file like the routing policy it actually is. The alternative is discovering, months from now, that you handed a competitor every citation in an entire engine because of a line nobody remembered writing.
Key takeaways
- A broad robots.txt block added to stop one scraper can silently exclude every AI answer engine that launched afterward, while Google indexing stays perfectly healthy.
- Nothing in your normal analytics flags the gap, so audit which named AI bots your current rules block rather than assuming access.
- Crescive's crawler analytics maps each AI bot against your live robots.txt and names exactly which engines a stale rule is turning away.
FAQ
How do I know if my robots.txt is blocking AI crawlers like GPTBot or PerplexityBot?
Open yourdomain.com/robots.txt and look for a User-agent: * line paired with a broad Disallow, which catches AI bots that don't have their own explicit exception. Then check your server or CDN logs for user-agent strings like GPTBot, PerplexityBot, and ClaudeBot. If a bot has zero hits or is matched by a disallow rule, it's being blocked. Crescive's crawler analytics automates this by flagging exactly which AI bots your current rules exclude and which rule is doing it.
Which AI crawler user-agents should I check for in my robots.txt?
The main answer-engine crawlers to check are GPTBot and OAI-SearchBot (OpenAI/ChatGPT), PerplexityBot (Perplexity), ClaudeBot and Claude-Web (Anthropic/Claude), and Google-Extended (Google Gemini and AI Overviews). If any of these are caught by a wildcard block, add explicit Allow rules for them above your catch-all so they can reach your pages.