Eighteen months after llms.txt became a recurring topic in SEO circles, adoption sits at roughly 8.7% of the world’s top 1,000 sites, and the harder finding is what those sites got for it. AI search crawlers — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — are overwhelmingly not fetching the file and crawling HTML directly instead. No major AI company has publicly committed to reading or acting on it in production. If you’re deciding whether to spend time on this in 2026, that decision should be based on what it actually does, not on what the format’s name implies.
What llms.txt Is
It’s a Markdown file at the domain root that lists a site’s most important pages with one-line descriptions — a curated index, not a crawling directive and not a schema format. Nothing about the spec forces any crawler to read it, respect it, or prioritize the pages it lists. It works the way a sitemap.xml file’s content works for a human skimming it, except no search engine or AI system is contractually or technically obligated to consume it the way they are (loosely) obligated to consider a sitemap.
What the Evidence Supports
| Claim | What the Data Shows |
|---|---|
| “llms.txt improves AI Overview citations” | Not supported — major AI search crawlers are not observed fetching the file in typical operation |
| “llms.txt is required for AI visibility” | Not supported — no AI company has committed to reading it, and sites without one are cited in AI answers routinely based on normal page content |
| “llms.txt helps AI coding tools understand a codebase or docs site” | Supported — the IDE-agent ecosystem (Claude Code, Cursor, and similar tools) does use it when developers point an agent at a project or docs site |
| “llms.txt costs little to add” | Supported — it’s a static Markdown file, typically under a day of work to generate and maintain for a site of moderate size |
Who Should Ship One
- Developer-tool and API documentation sites — the audience most likely to point an AI coding assistant directly at the domain, where the practical benefit is real and immediate.
- Sites already producing a WordPress-generated or plugin-generated version at near-zero marginal cost — if the tooling already exists, there’s little reason not to ship it.
- Sites making an explicit bet on future crawler adoption — a defensible, low-cost hedge, as long as it’s framed internally as a hedge and not sold as a current SEO or GEO win.
Sites that should deprioritize it: anyone treating this as a substitute for the fundamentals that do measurably affect AI citation — clean HTML structure, clear entity markup, content that directly answers a specific query, and genuine authority signals. Time spent hand-crafting llms.txt instead of fixing a page’s actual clarity is time misallocated.
If You Do Ship One
Keep it accurate and low-maintenance: link only to pages you’d want an AI coding tool or a developer skimming the file to actually land on, write descriptions that are genuinely distinct from each other (not padded restatements of the title), and regenerate it as part of your existing publishing pipeline rather than maintaining it by hand as a separate task that quietly goes stale.
