AI engines can only cite pages their crawlers can reach. Allow OAI-SearchBot, PerplexityBot, and others in robots.txt, using noindex only as a chosen opt-out.
AI engines can only cite pages their crawlers are allowed to reach, so blocking those crawlers in robots.txt quietly removes you from AI answers. The fix is to allow the AI crawlers you want — OAI-SearchBot, PerplexityBot, and the others — while using noindex only as a deliberate way to keep specific pages out. This guide covers which bots to allow and how to control access without cutting yourself off.
If a crawler cannot fetch your page, the engine behind it cannot read, index, or cite your content. A single overzealous robots.txt rule can make your whole site invisible to an AI engine, even when your content is excellent.
Many sites block AI bots by accident — a broad disallow rule, a security tool, or a default template that predates AI crawlers. The result is the same: competitors get cited and you do not.
Different engines use different user agents, and some use more than one — often a separate crawler for training versus live search retrieval. The common ones to know:
Because the landscape shifts, treat any list as a starting point and check your logs for which agents actually request your pages.
In robots.txt, you allow a bot by not disallowing its user agent — or by writing an explicit allow rule. The safest posture for AI visibility is to permit the search-and-answer crawlers on your public content.
A simple, permissive pattern looks like this:
Avoid a blanket "disallow all" that catches AI bots as collateral damage. If you must block, block specific paths, not everything.
Robots.txt controls crawling; a noindex directive controls whether a page should appear in results. Use noindex when you genuinely do not want a page surfaced — thank-you pages, internal utilities, duplicate content — not as a general privacy blanket.
Remember that a page blocked in robots.txt cannot be crawled, so an engine may never even see a noindex tag on it. If you want a page excluded cleanly, allow the crawl but apply noindex, rather than blocking the crawl outright.
Read your live robots.txt and confirm no rule accidentally disallows an AI user agent. Then check server logs or a crawler-access tool to see which bots are reaching your pages and which are being turned away.
Also watch for non-robots blocks: firewalls, rate limiters, and bot-management tools that challenge or reject non-browser requests. These can block AI crawlers even when robots.txt is perfectly permissive.
TrueCite's crawler check and LLMs.txt scanner report which AI bots can reach your site and flag rules or defenses that are shutting them out. That turns a guessing game into a concrete list of fixes.
Crawler access is easy to break by accident long after you first set it up. A new security tool, a CDN rule, a migrated robots.txt, or a template change can quietly start blocking AI bots.
Because these regressions are silent — nothing errors, you just stop appearing — build a periodic check into your routine. Re-read your live robots.txt, scan server logs for AI user agents, and confirm the bots you want are still getting through.
Pay special attention around infrastructure changes: launches, migrations, and new bot-management or firewall deployments are the usual culprits. Catching a block early is far cheaper than discovering months later that an engine stopped citing you because it could no longer fetch your pages.
Being citable starts with being reachable. Allow the AI crawlers you want in robots.txt, keep disallow rules narrow, reserve noindex for pages you truly want excluded, and verify with logs and a scanner that the right bots are actually getting in.
Using TrueCite? See the LLMs.txt Scanner docs →