LLMs.txt Scanner
The LLMs.txt Scanner checks whether your site has an llms.txt file, whether major AI crawlers are allowed access via robots.txt, and whether your sitemap is accessible. It returns an overall readiness score and specific fixes. Available on Lite and above.
What is llms.txt?
llms.txt is a plain text file at yourdomain.com/llms.txt that tells AI engines which pages to prioritize when learning about your brand. Think of it as robots.txt, but specifically designed for AI crawlers. Having a well-configured llms.txt improves the probability that AI engines accurately represent your brand in their answers.
Score breakdown
llms.txt present and valid
34 ptsllms-full.txt present
14 ptssitemap.xml accessible
10 ptsAI crawlers allowed (robots.txt)
Up to 42 pts (distributed across tracked AI bots)AI crawlers checked
TrueCite checks your robots.txt for the following AI crawler User-agents: GPTBot, ClaudeBot, PerplexityBot, GoogleBot, Amazonbot, and others. Any blocked crawler reduces your readiness score.
Running the scanner
Go to Dashboard → LLMs.txt tab
Navigate to the LLMs.txt section in your dashboard sidebar.
Click Scan
TrueCite checks your domain automatically using the website URL from your business profile.
Review results
Results show which AI crawlers are allowed or blocked, whether your llms.txt and sitemap exist, and your overall readiness score.
Follow fix recommendations
Each failing signal includes a specific fix. Blocked crawlers show the exact robots.txt line to add.
Fixing blocked AI crawlers
# Add to your robots.txt to allow AI crawlers:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Never block AI crawlers if you want AI engines to cite your content. Blocked crawlers are one of the most common reasons a brand never appears in AI-generated answers despite publishing high-quality content.
Scan history is saved so you can track your readiness score over time. Most sites go from ~20 to 80+ with three changes: add llms.txt, add sitemap.xml, and unblock AI crawlers in robots.txt.
Last updated: June 2026 — TrueCite documentation is updated with every product release.
Can't find what you're looking for?
Contact support →