GoogleProbably not. On the measured web, 2.10% of mobile sites publish a valid llms.txt file, and 39.6% of those files were generated by one SEO plugin’s default (Web Almanac, 2025). All the shares below come from HTTP Archive’s July 2025 crawl of 16.2 million sites, 15.4 million of them mobile.
Does your business need an llms.txt file?
Almost certainly not yet. The format is a proposal on its second version, no Google service uses it, and Google’s own checker does not count its absence against you.
The spec describes itself as a proposal to standardise on an /llms.txt file, and says it is v2, updated after two years of adoption (llms.txt spec, v2).
The Web Almanac’s SEO chapter states that Google has often said it does not use llms.txt, and that no Google service currently does.
The same chapter notes that Anthropic has taken a lead on the format and publishes a full Markdown version of its own documentation.
When the file is missing, Google’s own Lighthouse marks its llms.txt audit not applicable instead of failing it (Lighthouse 13.5.0 source).
What is an llms.txt file supposed to do?
Hand an assistant a plain Markdown map of your site instead of making it parse your HTML. Only one part of the file is required.
- 01An H1 with the name of the siteThe spec calls this the only required section. Everything after it is optional, which is why so many published files are nearly empty.
- 02A blockquote summaryA short description carrying the key information needed to understand the rest of the file.
- 03H2 sections listing linksEach entry is a Markdown link, a name and a URL, with optional notes after a colon.
- 04An "Optional" sectionBy convention, secondary links an agent can skip when it needs a shorter context (llms.txt spec, v2).
Who actually has one, and where did it come from?
Mostly a plugin, not a decision. The Almanac identified these files by the comments two WordPress SEO plugins leave by default, and cautions that their presence may not be a conscious act.
Web Almanac, 2025 · mobile sites with a valid file · the third bar is our subtraction, not theirs
View data
| Item | Value |
|---|---|
| All in One SEO | 39.6% |
| Yoast SEO | 3.6% |
| Everything else | 56.8% |
Does anything actually read it?
Not Google. The Web Almanac’s SEO chapter says no Google service uses the file, Anthropic publishes one, and Google’s own tool grades a file that exists rather than its presence.
The SEO chapter is direct about it. No Google service currently uses llms.txt, and the chapter calls the format a point of controversy in the industry.
The same chapter credits Anthropic with taking a lead on the format, which is a stance by one vendor, not a measured ranking effect.
Lighthouse 13.5.0 ships an llms.txt audit inside a category called Agentic Browsing, whose own description says it is still under development and subject to change.
If the request for the file returns a client error, the audit returns a passing score and marks itself not applicable (Lighthouse 13.5.0 source).
H1 header
missingpresent
The audit tests for a line that starts with a single hash and has text after it.
Markdown links
noneat least one
It looks for a single link, in name and URL form, anywhere in the file.
Length
under 50 charactersover 50
Below that threshold the audit reports the file as suspiciously short.
View data
| Metric | Day zero → target |
|---|---|
| H1 header | missing → present |
| Markdown links | none → at least one |
| Length | under 50 characters → over 50 |
What are site owners doing about AI crawlers instead?
Naming them in robots.txt, which more sites do every year. gptbot is named on 4.2% of mobile sites, up from 2.7% a year earlier.
Web Almanac, 2025 · 15.4 million mobile sites, July 2025 crawl
View data
| Item | Value |
|---|---|
| petalbot | 4.4% |
| gptbot | 4.2% |
| claudebot | 3.4% |
| ccbot | 3.2% |
| google-extended | 3% |
| amazonbot | 3% |
| perplexitybot | 2.7% |
| chatgpt-user | 2.5% |
The pattern is rank, not industry: the largest sites write these rules and the long tail mostly does not.
Web Almanac, 2025 · Generative AI chapter, robots.txt directives by site rank
View data
| Item | Value |
|---|---|
| Top 1k | 20.9% |
| Top 10k | 12.1% |
| Top 100k | 6.1% |
| Top 1M | 3.6% |
| Top 10M | 4% |
On popular websites the Almanac found these directives far more common, with nearly all of them disallowing access. That reading covers popular sites only.
The Generative AI chapter is explicit: adherence to robots.txt is voluntary and does not strictly enforce access control.
OpenAI’s bot documentation describes separate crawlers for model training, for the index behind ChatGPT’s search, and for fetches a person asked for. Paraphrased from that documentation, not quoted.
These are the user agent names site owners wrote into a file, not a measurement of what any crawler did.
So what should a service business do instead?
Two things that are documented: be readable by a crawler, and keep your business facts in text where both search and an assistant can lift them.
- 01Confirm a crawler can reach and render your pagesNothing downstream works if this one fails. See what Google sees when it crawls a contractor’s site.
- 02Keep hours, service area and phone as textInside an image, those facts are not text any reader can lift, human or machine.
- 03Keep your Google Business Profile currentIt is the record both search and assistants quote for a local company. See what makes a profile show up in the map pack.
- 04If a plugin already wrote an llms.txt, make it validAn H1, at least one link, more than 50 characters. Deleting the file is not the fix.
If a plugin already wrote the file, make it valid and move on. Nothing measured says it earns you an answer.
- 01Make the pages crawlable
- 02Put the facts in text
- 03Keep the profile current
- 04Then look at llms.txt
Questions owners ask
Is llms.txt an official standard?
No. The spec calls itself a proposal and is on its second version. The only required part of the file is an H1 with the name of the site; everything after that is optional.
Will my site score worse without one?
Not in Google’s own checker. When the file is missing, Lighthouse 13.5.0 marks the llms.txt audit not applicable instead of failing it. Absence is not a penalty.
I already have one. Is it valid?
Lighthouse checks three things: an H1 header, at least one Markdown link, and more than 50 characters of content. Fix those before anything else, and do not delete the file.
Does blocking gptbot hide me from AI answers?
It is a request, not a lock. The Web Almanac notes that following robots.txt is voluntary and does not strictly enforce access. On the top 1,000 sites, 20.9% name gptbot at all.
Sources
- HTTP Archive, Web Almanac 2025, SEO chapter (July 2025 crawl, 15.4 million mobile sites; read from the chapter's source text, commit 46992bf, October 24, 2026) (2025)
- HTTP Archive, Web Almanac 2025, Generative AI chapter (robots.txt figures measured on the ~12.9 million sites that chapter analyzed) (2025)
- HTTP Archive, Web Almanac 2025, Methodology (16,213,084 sites, 15,426,398 of them mobile; CrUX release of July 11, 2025) (2025)
- Google Lighthouse 13.5.0, llms.txt audit and Agentic Browsing config (open-source code, core/audits/agentic/llms-txt.js, commit 1a4ffa4, read October 24, 2026) (2026)
- The /llms.txt file, v2 (the proposal's own source, AnswerDotAI/llms-txt, commit 3700a0a) (2026)
- OpenAI bot documentation (paraphrased, not quoted; the host did not open from this session) (2026)
The first meeting starts with findings, not promises.The free Business Health Report reads your business and shows you what it found, in days.
Get Your Free Business Health Report