Everyone’s Adding an llms.txt File. I Checked Whether It Actually Does Anything.

If you run content or SEO right now, someone has probably told you that you need an llms.txt file: a vendor, a LinkedIn post, an audit tool flagging its absence in red. The pitch is seductive because it sounds like robots.txt for the AI era: one small file, uploaded once, and suddenly ChatGPT and Perplexity cite you more.

I build AEO/LLMO strategy for a living, so before recommending the fashionable file to anyone I went looking for citation data rather than vendor promises, and the evidence turned out to be unusually one-sided.

What is llms.txt and what does it claim to do?

An llms.txt file is a proposed convention: a Markdown file placed at your domain root (yoursite.com/llms.txt) that gives large language models a curated, machine-readable summary of your site. It was proposed in September 2024 by Jeremy Howard, the technologist behind fast.ai, to hand AI systems clean content instead of forcing them to parse cluttered HTML (Ranqo, 2026).

The claim attached to it in marketing circles goes further than the proposal ever did: that the file improves how often AI systems cite you, and that skipping it means falling behind. That’s the claim worth testing, because the file itself is trivial to make; the question is whether anything reads it.

One definitional point is important before the data: llms.txt is a community convention, not a standard; no standards body backs it, and there’s no enforcement mechanism, so AI companies adopt it, or don’t, entirely on their own terms (Derivatex, 2026).

Does llms.txt work? What the citation data shows

Across every large-scale study I could find, the presence of an llms.txt file has no measurable effect on AI citations, and the crawlers it’s built for barely fetch it. The evidence stacks up from three independent directions:

1. The correlation study. SE Ranking analysed roughly 300K domains, testing whether having an llms.txt file correlated with how often a domain was cited across major LLM responses, using both standard statistical tests and an XGBoost predictive model. Their conclusion: no significant correlation, and removing the file as a variable actually improved the model’s accuracy, meaning the file was adding noise rather than signal (SE Ranking, 2025). Adoption in their dataset sat at 10.13%, and among the 50 most-cited domains, exactly one had the file (Derivatex, 2026).
2. The server-log studies. Ahrefs analysed 137K domains and found that 97% of valid llms.txt files received zero requests in an entire month, no bots, no humans, nothing, and that AI bots made zero requests for files that didn’t exist, meaning they never proactively look for one (Ahrefs, 2026). OtterlyAI ran the single-site version of the same test: over a 90-day window, out of 62,100 AI bot visits to their test domain, 84 requests touched the file, about 0.1% of AI crawler traffic, roughly the attention an average PDF gets. They subsequently removed the check from their own GEO audit because it was distracting clients from work that moves citations (OtterlyAI, 2026).
3. The platforms themselves. Google’s guidance on optimising for generative AI features, in a section literally titled “mythbusting,” states that site owners don’t need machine-readable files, AI text files, or Markdown to appear in generative AI search (Greadme, 2026). Google’s John Mueller went further on Reddit:

“To me, it’s comparable to the keywords meta tag.” (WebYes, 2026)

The keywords meta tag was where site owners typed what they wished their site ranked for, and search engines stopped reading it because self-declared relevance is worthless; that’s the comparison Google’s own search team reaches for. As of early 2026, no major AI provider, OpenAI, Google, Anthropic, Meta, or Mistral, has publicly committed to reading the file in production answer systems (Derivatex, 2026).

Here’s the whole evidence base side by side:

Study	Scale	What it measured	Finding
SE Ranking (Nov 2025)	~300K domains	Correlation between llms.txt and AI citation frequency.	No significant correlation; removing the variable improved model accuracy.
Ahrefs (2026)	137K domains	Actual requests to llms.txt files in server logs.	97% of files got zero requests in a month; AI bots never look for missing files.
OtterlyAI (2026)	1 domain, 90 days	Share of AI bot traffic touching llms.txt.	0.1% of 62,100 AI bot visits; on par with a PDF.
Google (2025-2026)	Official guidance	Whether the file is needed for AI features.	Explicitly not needed; Mueller compares it to the keywords meta tag.

One caveat the loudest takes skip: the Ahrefs and Wix data show the file does get crawled and indexed in some volume, and adoption keeps climbing anyway, 28% in Ahrefs’s tech-skewed sample (Ahrefs, 2026). Being fetched occasionally is not the same as being used as a citation signal, and no study has found the second part.

Why hasn’t llms.txt been adopted by the AI companies?

The structural reason is the same one that killed the keywords meta tag: llms.txt is a self-authored description of your own site, and self-declared signals are the easiest ones to game. AI citation systems already evaluate sources using signals that are harder to fake, publication authority, third-party corroboration, how often credible sources mention you, and a curated file you wrote about yourself sits at the bottom of that trust hierarchy, not the top (AuthorityTech, 2026).

There’s also a simpler operational reason: the retrieval systems behind AI answers run on existing search infrastructure, crawling HTML, ranking pages, pulling passages, and Google has said its generative AI features run on core Search systems (Contentful, 2026). Maintaining a second ingestion path for a file most sites lack, and that anyone can fill with whatever they like, is a cost with no benefit to the provider, so adoption stalls.

Where should the effort go instead?

If the goal is being cited by AI systems, the levers with actual evidence behind them are the ones I’ve built my whole AEO/LLMO framework around, and none of them is a root-directory file.

• Substance a model can’t get elsewhere. Attributed statistics, named sources, original positions, etc. The Princeton GEO study found adding attributed statistics lifted AI visibility by around 41% and citing external sources by up to 115% for lower-ranked content (Princeton GEO study, KDD 2024). That’s the size of lever the file was hoped to be and measurably isn’t. I’ve written about why structure without substance stays invisible, and this file is the purest example of the pattern: it’s structure about structure.
• Off-site presence and third-party mentions. The citation systems weigh who talks about you more heavily than what you say about yourself. SE Ranking’s own follow-up work found domains with heavy Reddit brand mentions averaged nearly four times the ChatGPT citations of domains with minimal presence (Contently, 2026), which is a brutal contrast with the zero effect their 300K-domain study found for the file.
• Extractable, structured answers on the pages themselves. Front-loaded answers, question-format headings, self-contained sections, comparison tables; the crawlers that skip llms.txt are reading your HTML directly, so the HTML is where the optimisation belongs. Auditing an existing library against exactly those checks is what my AI readiness audit covers, and it will do more for citations than any file at your root ever will, because it improves the thing that actually gets retrieved. Whether any of it worked shows up in citation tracking, not in an audit tool’s checkbox, which is the same measurement discipline I apply to content metrics generally.

When is llms.txt still worth adding?

There are two honest cases for it, and neither is citations.

1. The first is developer documentation. AI coding assistants like Cursor and Claude Code fetch external docs in real time, and for API products a curated llms.txt pointing at canonical documentation demonstrably helps those tools generate working code instead of hallucinated endpoints, which is why Anthropic, Stripe, Vercel, and Cloudflare all ship one (Limy, 2026). Notably, in the Ahrefs log data the second-biggest named AI fetcher of these files was Claude Code, a coding tool, not a search bot (Ahrefs, 2026). If you sell to developers, the file is a developer-experience feature and worth thirty minutes.
2. The second is cheap insurance. The file costs almost nothing to create, carries little risk if you keep it accurate and don’t spawn duplicate-content Markdown copies of every page, and if a major provider ever does commit to reading it, you’re already there. That’s a defensible reason to add one, as long as everyone involved is honest that it’s a lottery ticket, not a strategy, and my own content prioritisation logic would score it: impact currently near zero, effort near zero, so fine as a fifteen-minute task, indefensible as a project.

The verdict

Don’t buy llms.txt as an AI-visibility lever, because the citation data says it isn’t one: the biggest correlation study found no effect, the server logs show almost nothing reads the file, and the companies that would have to read it have declined or stayed silent. Add one in fifteen minutes if you have developer docs or want cheap insurance, then spend the real budget on substance, third-party presence, and extractable page structure, the things the evidence says actually earn citations. If a vendor is charging meaningful money to implement the file, that tells you more about the vendor than the file, and the same evidence-over-hype standard applies to the next fashionable checkbox, because AEO built on weak foundations doesn’t get rescued by one.