Robots.txt Generator with AI Bot Presets

How robots.txt rules work

A robots.txt file sits at the root of a host and holds groups of rules. Each group starts with one or more User-agent lines followed by Disallow and Allow paths. The format is the Robots Exclusion Protocol, standardized as RFC 9309, and Google documents how it reads it:

• A crawler follows the group that names it most specifically and falls back to User-agent: * only when no group names it. Once you add a group for a bot, that bot ignores your * rules, private paths included.
• When rules conflict, the most specific rule (the longest path) wins. When an allow and a disallow rule are equally specific, Google uses the less restrictive one.
• User-agent names are case-insensitive, but paths are case-sensitive: /Admin and /admin are different rules.
• * matches any run of characters and $ marks the end of a URL, so Disallow: /*.pdf$ blocks every PDF.
• Sitemap lines must be full URLs, including the protocol and host. They are not tied to any user-agent group, and you can list several.
• Google supports only user-agent, allow, disallow and sitemap. It ignores crawl-delay and stops reading after the first 500 KiB.

The right robots.txt rules for AI crawlers

AI companies run separate bots for separate jobs, which is why the generator groups them:

• AI search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) fetch pages so an assistant can cite and link to them. Block them and you drop out of those answers.
• AI training and open datasets (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and others) decide whether your content is collected for model training. OpenAI, Anthropic, Google and Apple document these separately from their search products, so blocking them does not take you out of AI search.
• User-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User and others) open a page because someone asked. Several operators say these may ignore robots.txt.

For most startups the recommended preset is the sensible default: you stay findable in AI answers, which is where the citations and clicks come from, and you keep a say over training. To block a single training bot, the group is two lines, User-agent: GPTBot and Disallow: /. Google's AI Overviews and AI Mode use Googlebot, so there is no robots.txt rule that removes you from them without removing you from Google Search.

Blocking crawling is not blocking indexing

Disallow stops compliant crawlers from fetching a URL, but Google says it can still index a blocked URL, without its content, if other pages link to it. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header, or put it behind a login.

Do not block the CSS and JavaScript files your pages need. Google says that if leaving them out makes a page harder for its crawler to understand, you should not block them.

Common robots.txt mistakes

• Shipping a staging file. User-agent: * with Disallow: / blocks every crawler from the whole site.
• Listing secret URLs. Anyone can read robots.txt, so it points people straight at the paths you list. Protect private pages with a login.
• One file for every subdomain. Rules apply only to the host, protocol and port the file is served from, so blog.example.com needs its own file.
• A relative sitemap URL. Google requires a fully qualified URL, so Sitemap: /sitemap.xml is not valid. Write the full address.
• Putting noindex in robots.txt. Google ignores it there. Use the robots meta tag or header instead.
• Copying a "block every AI bot" list. Many include search bots, which removes you from ChatGPT search, Claude or Perplexity answers you may want.

Where to put the file and how to test it

Upload robots.txt to the top-level directory of each host, so it loads at https://yourdomain.com/robots.txt. Google says it generally caches the file for up to 24 hours, and other crawlers refresh on their own schedules. In Google Search Console, the robots.txt report shows the copy Google fetched and lets you request a recrawl after an important change.

Then run the AI crawler checker on your home page and a few key pages to confirm each bot sees what you intended. If you also want a map of your best pages for AI tools, draft an llms.txt with the llms.txt generator.