Free AI Crawler Checker: Can ChatGPT, Claude and Perplexity Crawl Your Site?

What the AI crawler checker tests

The checker requests three things from your site with its own clearly named user agent, the same way a crawler would before it reads a page:

• /robots.txt on the host you entered (or the host the page redirects to, because crawlers follow redirects). Each AI bot's access is worked out with the rules of RFC 9309, the robots.txt standard: a bot obeys the group that names it, falls back to the User-agent: * group only when no group names it, and the longest matching path wins.
• The page itself, for its robots meta tags and the X-Robots-Tag header, such as noindex, nosnippet and noarchive.
• /llms.txt, to see whether the optional file exists.

How the robots.txt request goes matters too. If the file returns 404, crawlers may crawl everything. If it returns a server error or cannot be reached, the standard tells crawlers to treat the whole site as blocked, and Google says it stops crawling for the first 12 hours of such errors.

AI search, AI training and user fetches are separate bots

Most AI companies run one bot per job, and each obeys only its own robots.txt group. That is why the results come in three groups:

• AI search and answers: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot and others fetch pages so an assistant can cite and link to them. Google's AI Overviews and AI Mode are part of Google Search, so they follow your Googlebot rules.
• AI training and open datasets: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and others decide whether your content is collected for model training or public datasets. Google-Extended and Applebot-Extended are control tokens, not separate crawlers.
• User-triggered fetches: ChatGPT-User, Claude-User, Perplexity-User and others open a page because a person asked. OpenAI, Perplexity, Meta, Amazon and Google say some of these may not follow robots.txt, because a user started the request.

So blocking a training bot does not remove you from AI search. OpenAI documents that GPTBot and OAI-SearchBot are controlled separately, Anthropic documents ClaudeBot and Claude-SearchBot as separate bots with separate opt-outs, and Google says Google-Extended does not affect inclusion or ranking in Google Search.

How to read the results

• Allowed: no rule stops the bot from fetching this page, and the rest of the site is open by default.
• Allowed with limits: this page is open, but the bot's rules close the home page or the site in general, and only the paths you allow are open.
• Blocked: a Disallow rule matches this page. The row shows the line number, so you can find it in your file.

A bot your file does not name follows your User-agent: * rules, and the row says so. Apple is the exception worth knowing: Apple says Applebot follows your Googlebot rules when you do not mention Applebot.

Rows for AI training bots are marked as information rather than pass or fail, because allowing or blocking training is your call. Rows for fetchers that may ignore robots.txt are flagged too, so you do not rely on a rule they might skip.

Common mistakes that block AI search by accident

• A leftover staging rule. User-agent: * followed by Disallow: / blocks every crawler, search engines and AI search included.
• A copied "block AI bots" list. Many lists mix training bots with search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, which removes you from those assistants' answers.
• Forgetting that a named group replaces the general rules. Once a bot has its own group it ignores your User-agent: * group, so it can lose your private-path rules or end up blocked when you meant to allow it.
• A robots.txt that errors or sits behind a bot challenge. Crawlers that cannot read the file must treat the whole site as blocked.
• noindex or nosnippet on pages you want cited. Google says a page must be indexed to appear as a link in AI Overviews and AI Mode, and that nosnippet keeps the text from being used in them.
• Relying on noai meta tags. They are not part of any standard, and the major AI crawler documentation does not mention them. Use robots.txt rules instead.

How to fix a blocked AI crawler

1. Find the line the checker names in your robots.txt.
2. If the bot has its own group, delete its User-agent line (it then follows your rules) or change the Disallow. If your group blocks it, open that path or give the bot its own group with the private paths you still want closed.
3. Keep a Sitemap: line with your sitemap's full URL.
4. Upload the file to the root of the host and run the check again.

Changes are not instant. Google says it generally caches robots.txt for up to 24 hours, OpenAI and Perplexity mention about 24 hours, and DuckDuckGo says up to 72 hours. If robots.txt allows a bot but your host or CDN blocks it, change the bot settings there: Cloudflare, for example, offers a managed robots.txt and AI Crawl Control. You can also build a fresh file with the robots.txt generator, and read how to get recommended by ChatGPT for what comes after access.