Your robots.txt controls which AI crawlers can read your site. Here is how to configure directives for the major AI bots and avoid common mistakes.
Your robots.txt file controls which crawlers may read your site, and that now includes the AI crawlers that feed answer engines. By adding user-agent blocks for bots like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, you decide which AI systems can access your content — and to be cited in AI answers, you generally need to allow the crawlers behind the engines your buyers use.
This guide is about the configuration itself: the directives, the patterns, and the mistakes to avoid. The broader question of whether to allow or block AI crawlers is a strategy decision; here the focus is getting the file right once you have decided.
A robots.txt file is a set of blocks, each naming a user-agent and listing rules that apply to it. For AI, you add blocks for the specific AI crawlers you want to control.
The key detail is specificity: a named user-agent block for an AI bot overrides the wildcard for that bot, so you can treat individual AI crawlers differently from the general default.
The simplest configuration is permissive: a wildcard user-agent with an empty Disallow allows every crawler, AI bots included. Many sites run effectively allow-all and are citable by default because nothing blocks the AI crawlers.
If you want finer control, you add explicit blocks for named AI bots rather than relying only on the wildcard. That lets you, for example, allow the crawlers behind the engines your buyers use while handling others individually. The point is that AI crawlers are controlled per user-agent, so a selective setup is just a matter of adding the right named blocks.
A few robots.txt mistakes quietly harm AI visibility.
Most of these come from treating AI crawlers as a single group rather than distinct, individually named agents.
It helps to see robots.txt alongside llms.txt, because they play complementary roles for AI. Robots.txt controls access — which crawlers may read your site at all. Llms.txt, by contrast, is a guide — a file that points AI systems toward your most important content and summarizes it. One is a gate; the other is a signpost.
The practical takeaway is that allowing AI crawlers in robots.txt is the prerequisite, and llms.txt is the enhancement layered on top. There is little value in a detailed llms.txt if your robots.txt blocks the crawlers that would use it, and conversely, permissive robots.txt with no llms.txt leaves engines to find their own way around your site.
Handled together, they give AI systems both permission and direction: robots.txt lets the right crawlers in, and llms.txt helps them focus on what matters. Reviewing the two as a pair, rather than in isolation, is the more complete way to manage how AI accesses and understands your site.
Because a robots.txt mistake fails silently — nothing errors, you simply stop being crawled — it is worth verifying rather than assuming. Review the file for Disallow rules that touch AI bots, confirm each named AI user-agent is handled as you intend, and check that your important pages are reachable.
TrueCite’s LLMs.txt Scanner and crawler checks surface AI bot access issues, showing whether the major AI crawlers can reach your site so a misconfigured directive does not quietly keep you out of AI answers. Getting robots.txt right for AI is mostly careful, per-bot configuration — and confirming it does what you meant.
Check your AI crawler access with TrueCite — 7-day free trial, no card required.
Using TrueCite? See the LLMs.txt Scanner docs →