robots.txt and AI Bots: Should You Allow or Block AI Crawlers?

The AI Crawler Question Every Brand Needs to Answer

Your robots.txt file tells web crawlers what they can and cannot access. Traditionally, it was about search engine bots. Now, it is also about AI crawlers from ChatGPT, Perplexity, Claude, and others.

The decision to allow or block AI crawlers has direct consequences for your AEO. Here is what you need to know.

The Major AI Crawlers

Why You Should Allow AI Crawlers (In Most Cases)

If you want AI engines to cite your brand, those engines need to be able to read your content. Blocking AI crawlers prevents:

• ▸ChatGPT from learning about your products and recommendations
• ▸Perplexity from citing your FAQ pages and product descriptions
• ▸Claude from referencing your technical content
• ▸AI engines from updating their knowledge about your brand

The math is simple: if AI bots cannot crawl your site, AI engines cannot cite you. And if AI engines cannot cite you, you have zero presence in the AI research sessions that influence your buyers.

When Blocking AI Crawlers Might Make Sense

There are legitimate reasons to block specific AI crawlers:

• ▸Proprietary content — if you have valuable content you do not want used for AI training
• ▸Paywalled content — resources that require payment and should not be freely available to AI
• ▸Legal restrictions — content with licensing restrictions that prohibit AI training use
• ▸Competitive intelligence — if competitors use AI tools to mine your content for insights

If these apply to specific sections of your site, use path-specific robots.txt rules rather than blanket blocks.

How to Configure robots.txt for AI Visibility

Allow all AI crawlers (recommended for most brands):

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: FacebookBot
Allow: /

Allow AI crawlers but block training on specific content:

User-agent: GPTBot
Allow: /
Disallow: /private/
Disallow: /customer-data/

User-agent: Google-Extended
Allow: /
Disallow: /internal/

Block specific AI crawlers (use sparingly):

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

The AI Crawler Permission Score

TrueCite calculates an AI crawler permission score based on how many of the 9 major AI crawlers can access your site. The score affects your overall AEO health:

• ▸90-100% — All or nearly all AI crawlers allowed. Optimal for AEO.
• ▸60-89% — Most crawlers allowed. Minor gaps in specific engines.
• ▸Below 60% — Significant crawler blocks that will limit AI citations.

Beyond robots.txt: Additional AI Access Signals

Meta AI no-index tags

The `<meta name="robots" content="noai, noimageai">` tag signals to some AI systems not to use page content for training. Use selectively — if applied site-wide, it may reduce your AI citation potential.

Crawl-delay directives

If you have a high-traffic site and want to control AI crawler load:

User-agent: GPTBot
Allow: /
Crawl-delay: 10

Sitemap reference in robots.txt

Always include your sitemap in robots.txt — this helps all crawlers, including AI ones, discover your content:

Sitemap: https://yourdomain.com/sitemap.xml

Check Your AI Crawler Status

Use TrueCite's Crawler Check to see:

• ▸Which AI bots are currently allowed or blocked on your domain
• ▸Your AI crawler permission score
• ▸Specific robots.txt rules affecting each AI crawler
• ▸Recommendations for improving your crawler access configuration

[Check your AI crawler status with TrueCite →](/dashboard/crawlers)