What Are AI Crawlers?
Sharmeen SaleemI'm a Content Strategist at Wellows, an AI visibility platform that helps brands track how they're cited across ChatGPT, Gemini, Perplexity, and Google's AI features. I create the…Read Full BioWhat Are AI Crawlers?
Definition: AI crawlers are automated bots run by AI companies that fetch web pages to train models, to build the search indexes behind AI answers, or to retrieve a page when a user asks about it.
Each AI company usually runs more than one bot, and each bot has a different job. That matters because you can allow one and block another. A site can stay out of model training and still be eligible to appear in AI search results.
What Are the Main AI Crawlers?
Purpose falls into three types: training (collecting content for models), search index (building the index AI search draws on) and user fetch (retrieving a page because someone asked).
| Bot | Company | Purpose | Follows robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search index (ChatGPT search) | Yes |
| ChatGPT-User | OpenAI | User fetch | Not guaranteed. Requests are user-initiated, so robots.txt rules may not apply |
| PerplexityBot | Perplexity | Search index (Perplexity results) | Yes |
| Perplexity-User | Perplexity | User fetch | Generally no. Requests are user-initiated |
| ClaudeBot | Anthropic | Training | Yes |
| Google-Extended | Training and grounding for Gemini. A robots.txt token, not a separate crawler | Yes. It is a robots.txt token |
Bot names and rules change, so the “Checked” column shows when each row was last confirmed against the company’s own documentation.
What Does Blocking Each AI Crawler Cost?
The bots are independent, so each block has a different effect.
| If you block… | What happens |
|---|---|
| GPTBot | You signal that your content shouldn’t be used to train OpenAI’s models. It doesn’t control whether you appear in ChatGPT search, which OAI-SearchBot handles |
| OAI-SearchBot | Your site isn’t shown in ChatGPT’s search answers, though it can still appear as a plain navigational link |
| ChatGPT-User | Robots.txt rules may not apply to these user-triggered visits, so blocking may not stop them. It isn’t used to decide search inclusion |
| PerplexityBot | Your pages aren’t surfaced and linked in Perplexity’s search results |
| Perplexity-User | Perplexity says it generally ignores robots.txt for user-initiated requests, so a Disallow rule may not stop these fetches |
| Google-Extended | You opt out of Gemini training and grounding. Google says it has no effect on inclusion or ranking in Google Search |
GPTBot vs OAI-SearchBot: OpenAI treats these as separate settings. You can disallow GPTBot to keep content out of training and allow OAI-SearchBot to stay eligible for ChatGPT search. Blocking both takes you out of both.
How Do You Control AI Crawler Access?
Robots.txt is the main control. This example keeps content out of OpenAI training while staying eligible for ChatGPT and Perplexity search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Three things to know:
- Allow time for changes. OpenAI and Perplexity both say it can take about 24 hours for a robots.txt update to take effect.
- Robots.txt is a request. Firewalls, CDN bot rules and rate limits can block a crawler even when robots.txt allows it, so check those too.
- Confirm the bot is who it says. OpenAI and Perplexity publish IP ranges you can check requests against.
Tools for managing AI crawler access:
- Robots.txt: the baseline for every crawler that honors it.
- CDN or firewall bot controls: for example, Cloudflare’s AI Crawl Control, which lets you allow or block AI crawlers at the edge.
- Server logs: to see which bots are actually visiting and how often.
- A site audit: the Wellows Technical SEO Audit includes an LLM / AI Access check for whether AI bots can reach your site.
Once a crawler can reach your pages, they still need to be easy for AI systems to read. See AI-readable content for how to structure them.
Why Doesn’t AI Crawler Traffic Show Up in GA4?
Two reasons. Most AI crawlers request a page without running the JavaScript tag that GA4 relies on, so GA4 never receives a hit. And GA4 automatically excludes traffic from known bots. Server logs or a CDN dashboard show AI crawler visits that GA4 can’t. For how bot traffic affects GA4 reporting, see how to detect bot traffic in Google Analytics.
Conclusion:
AI crawlers aren’t one thing. Training bots, search bots and user-fetch bots do different jobs, and blocking the wrong one can remove you from AI search while doing nothing about training. Decide what you want from each type, set robots.txt to match, and check your logs to confirm it works.
Why is GPTBot traffic not showing up in my standard GA4?
What are the best tools for managing AI crawler access?
Does blocking GPTBot remove my site from ChatGPT answers?
Does robots.txt stop all AI crawlers?
Summarise this article
Share this article
About the Author
I'm a Content Strategist at Wellows, an AI visibility platform that helps brands track how they're cited across ChatGPT, Gemini, Perplexity, and Google's AI features. I create the… More about me