AI Crawler Checker: Can AI Bots Crawl Your Site?
Enter any URL to see which AI crawlers its robots.txt allows or blocks, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended. Every result shows the exact rule responsible, so you know what to change. Free, no signup needed.
Why AI Crawler Access Matters for SEO
More and more searches now end in an AI-generated answer rather than a list of links. ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews all build those answers from web pages, and they can only cite pages their crawlers are allowed to reach. If your robots.txt blocks the wrong bot, you can disappear from AI answers without ever noticing.
The opposite problem is just as common. Many site owners want to stop their content being used to train AI models, but block the wrong user-agent or leave out half the crawlers that matter. This checker shows you exactly where you stand for each one. To see how likely your content is to be cited once crawlers can reach it, run it through the GEO content checker.
How to Use the AI Crawler Checker
Enter a URL
Paste any page address. The tool fetches that site's robots.txt and checks the exact path you entered, so you can test a blog post, a category or the homepage.
Read the Verdicts
Each crawler is marked Allowed or Blocked, grouped into AI search, AI training and search engines, with the robots.txt line that decided it.
Check the Advice
The What This Means panel flags problems, such as blocking ChatGPT search while allowing training, and explains what each setup does.
Test Your Changes
Draft new rules in the Paste robots.txt tab and test them before you publish, or copy the ready-made rules to allow AI search and block AI training.
AI Search Crawlers vs AI Training Crawlers
Not all AI crawlers do the same job, and treating them as one group is the most common mistake in robots.txt files today.
AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI assistants can find them, quote them and link back to the source. Blocking them means you are unlikely to be cited in those answers. Related fetchers such as ChatGPT-User, Claude-User and Perplexity-User visit a page only when a person asks the assistant about it.
AI training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect content that may be used to train future AI models. Blocking them keeps your content out of that training without affecting whether you appear in search. Google-Extended and Applebot-Extended are not separate crawlers but controls: they tell Google and Apple whether content already crawled by Googlebot or Applebot can be used for AI training.
Search engine crawlers such as Googlebot and Bingbot matter most of all. Googlebot powers Google Search, AI Overviews and AI Mode, and Bing's index feeds Microsoft Copilot. Blocking either is almost always a mistake.
How Robots.txt Rules Are Matched
The checker follows the official robots.txt standard (RFC 9309), the same rules Google uses. A crawler first looks for a group that names it in a User-agent line. If it finds one, it obeys only that group and ignores the User-agent: * rules entirely. If no group names it, it falls back to the * group.
Within that group, the longest matching Allow or Disallow path wins, and if two rules are the same length, Allow wins. An asterisk matches any characters and a dollar sign marks the end of the URL. These details explain most surprising results: for example, adding a GPTBot group to allow one folder can accidentally unblock GPTBot everywhere else, because it stops reading the * rules. If a page is blocked by a redirect rather than robots.txt, the redirect checker will show you the full chain.
Related Tools
AI Crawler Checker - FAQs
What is an AI crawler?
An AI crawler is a bot run by an AI company that visits web pages to collect content. Some, like GPTBot and ClaudeBot, gather content that may be used to train AI models. Others, like OAI-SearchBot and PerplexityBot, index pages so AI assistants can find, cite and link to them when answering questions.
How do I block GPTBot in robots.txt?
Add a group for GPTBot to your robots.txt file: a line reading User-agent: GPTBot followed by Disallow: /. This asks OpenAI's training crawler to stay off your whole site. It does not affect OAI-SearchBot, which is what ChatGPT search uses to show and link to pages.
Will blocking AI crawlers hurt my Google rankings?
No, as long as you leave Googlebot alone. Google Search, AI Overviews and AI Mode all use Googlebot. Blocking Google-Extended only stops your content being used for Gemini model training and grounding, and does not change how you rank or appear in Google Search.
Should I block AI crawlers?
It depends on your goals. Blocking AI search crawlers means tools like ChatGPT search and Perplexity are less likely to cite or link to you. Blocking AI training crawlers keeps your content out of future model training without affecting search. Many publishers block training crawlers and allow search crawlers, which is the approach the snippet in this tool sets up.
Do AI crawlers actually follow robots.txt?
The major AI companies say their crawlers respect robots.txt, but it is a request rather than a technical barrier. Fetchers that act on a user's direct request, such as ChatGPT-User, may follow different rules from automated crawlers. If you need to guarantee a block, it has to happen at your server or firewall.
Why does the checker say it could not read my robots.txt?
Some sites block automated tools, including this one, while still allowing real crawlers. If that happens, open yoursite.com/robots.txt in your browser, copy the contents and use the Paste robots.txt tab. The results will be exactly the same.
Is this AI crawler checker free?
Yes. It is completely free with no signup, and you can check as many sites as you like.