llms.txt is a proposed web standard: a plain-text file, written in Markdown and placed at the root of your site (for example, polarisagency.com/llms.txt), that offers large language models a curated, machine-readable guide to your most important content. Think of it as a friendly welcome map for AI systems that are already exploring your site, pointing them towards the pages you most want them to understand.
Having been around in search for over two decades, I have watched plenty of proposed standards arrive with a wave of excitement, and llms.txt is one of the more interesting recent examples. It has generated a lot of discussion, a fair amount of confusion (clients and SEO/GEO community), and some genuinely useful clarity from Google in 2026. The POLARIS GEO agency we have developed this guide to set out what llms.txt actually is, what the evidence tells us, what Google has said, and where I believe it earns its place in a modern strategy.
/ Key takeaways
A low-effort, low-risk Markdown file that helps AI agents navigate your site efficiently once they have arrived. Research across roughly 300,000 domains shows adoption sits at around 10%, and there is currently no measurable link between having the file and how often AI systems cite your content. Google has been refreshingly clear that its real value lies in on-site agent navigation rather than discovery or ranking. Our recommendation is that it remains a sensible, forward-looking addition for the right sites, provided your expectations are set correctly.
What is llms.txt?
The concept is elegantly simple, you create a single Markdown file then upload it at your domain root, and use it to present a concise summary of your site alongside a curated list of the pages, documents, or resources you consider most valuable.
The format tends to follow a clean structure: a top-level heading with your site or brand name, a short summary in a blockquote, and then grouped links to your priority content. Many implementations also include a companion llms-full.txt file, which expands on the same idea with fuller content for models that can accept more context.
People often reach for the robots.txt comparison, and it is a helpful starting point because both files live at the root and both speak to automated visitors. It is worth holding that analogy loosely, though. Robots.txt governs crawler access and permissions, whereas llms.txt exists to guide and inform, therefore it is seen more as a signpost offered in good faith rather than a set of rules that a user agent has to follow.
NB: However, robots.txt based on our experience analysing server log files at present gets crawled over 150% more times than the llms.txt file.
What does an llms.txt look like?
A basic example gives you the shape of it quickly: a top-level heading with your company name, a short summary sentence, and then grouped sections such as Core documentation and Key resources, each listing a handful of links with a brief description after each one.
The appeal here is that it is human-readable, quick to produce, and easy to maintain. For anyone who has wrestled with complex technical implementations over the years, the low barrier to entry is genuinely refreshing (which usually means the impact would be low).
Losing clicks to AI-generated answers? Find out exactly how visible your brand is inside ChatGPT, Gemini and Perplexity before your competitors do.
What problem was llms.txt designed to solve?
To appreciate llms.txt properly, it helps to understand how AI systems handle content, because this is where the file potentially adds value.
Language models work within context windows, and processing content costs tokens. When an AI agent lands on a typical web page, it encounters a great deal of surrounding material: navigation, scripts, styling, boilerplate, and layout code that exists to serve human visitors. Parsing all of that to locate the substance is expensive in token terms and can be inefficient, this is why ensuring the page can be easily crawled, defined accessibility tree and logical DOM is critical for GEO success.
This brings us to the distinction that sits at the heart of the entire topic: the difference between crawl-time discovery and inference-time retrieval. Discovery is how a system learns your site exists in the first place. Retrieval at inference time is what happens once an agent is already engaging with your site and wants to work out what matters and where to find it. llms.txt is built for that second moment, to offer a low-token, pre-curated map to an agent that has already arrived, so it can find your best content quickly and cleanly.
Holding that distinction firmly is the single most useful thing you can do when evaluating llms.txt, and it explains almost everything about the guidance that has emerged since. It has little to no value in “ranking you in AI assistants” (AI doesn’t rank websites, SEO agencies that focus on measuring like traditional SEO is wrong – it cites recognition, recency and visibility on a probabilistic framework).
Does llms.txt actually work? What the data says
One of the most valuable contributions to this conversation is a large-scale study by SE Ranking, published in November 2025, which analysed close to 300,000 domains. It gives us real evidence to reason from rather than speculation.
The headline findings are that adoption stands at roughly 10%, so around one in ten sites has implemented the file, which places it some distance from the near-universal presence of robots.txt or XML sitemaps. Adoption is also spread fairly evenly across sites of all sizes, with high-traffic domains slightly less likely to use it than mid-tier ones. Most importantly, the study found no correlation between the presence of llms.txt and how frequently a domain was cited by AI models. When the researchers removed the llms.txt variable from their machine-learning model, its accuracy actually improved.
The practical reading is straightforward and resonates with my stance since day one. The evidence confirms that llms.txt is doing something other than driving discovery or citations today, which lines up precisely with its original design purpose. Once you accept what the file is for, the data stops being disappointing and starts being clarifying.
What Google has said about llms.txt (2026 update)
Google addressed this topic directly on its Search Off the Record podcast (episode 111, “Should I use markdown for my site?”, 15 June 2026), with John Mueller and Martin Splitt discussing markdown and llms.txt in detail. The full transcript is available via Google at goo.gle/sotr111-transcript, and Mueller’s comments (as quoted in coverage by Search Engine Journal) are worth reading closely because they are more nuanced than some of the headlines suggested.
On the original intent of the proposal, Mueller explained that he had spoken with one of its creators, and that the aim was “really not to create something that makes it easier for search engines or LLM systems to discover all of your content, but almost more that if an LLM already knows about your site and wants to find out what else is here, then that might be an approach.” He was candid that using it “as a way to optimize for Discovery by AI systems… doesn’t make any sense at all.”
On why the file sits outside the discovery process, he pointed to the question of trust. Because the file is the site owner’s own description of their content, an LLM “by design, can’t trust what is here as a way of differentiating between different websites.”
On the genuine use case, Mueller was encouraging about the on-site scenario. “If someone is already on your website, maybe some kind of automated system is helpful,” he said, describing how an agent already present on a site could look around to work out how to complete a task. He noted that emerging approaches such as WebMCP aim to do “something similar” for agentic interactions.
He also reaffirmed that the familiar SEO fundamentals remain anchored in your pages, observing that finding a website “is almost going to be completely bound to HTML pages and normal web pages.”
It is worth addressing the point that amused a lot of people, which is that Google publishes llms.txt files on its own properties (along with Markdown) while advising the industry against relying on them for ranking. Once you apply the discovery-versus-navigation distinction, the two positions sit together comfortably. Publishing a file to help agents navigate a site is entirely consistent with saying it will not influence discovery or rankings. Both statements describe the same file doing the same modest, useful job.
Thoughts on this topic have been well debated since 2024, my position remains unchanged.
“A lot of the businesses I speak with have read a blog or two and arrived convinced that llms.txt is the holy grail that unlocks AI search performance. It is a simple file and it should be treated as one, at its best it future-proofs your site and helps agents avoid burning through tokens. That is a genuine, if modest, benefit. At its most costly, it becomes a distraction that pulls attention away from the activity that actually moves the needle in AI search.
The simple rule of thumb – when something can be done quickly, easily, with very little effort, and is self-validated, it will usually deliver very little business gain on its own. The real value within technical GEO sits with the harder, higher-leverage work: clean crawl paths, server-rendered content, strong entity signals, and structured data that earns citations”.
What about the other AI providers?
Google is one search engine in a much larger field, and llms.txt matters most to the systems that actually read it: ChatGPT, Claude, Perplexity, and a growing list of agents. So where do the others stand?
The picture is consistent, none of the major AI search providers have confirmed it uses llms.txt as a discovery or citation signal, and that holds for OpenAI, Anthropic, and Perplexity as much as for Google. OpenAI points publishers towards allowing its crawlers through robots.txt, with no sign that llms.txt affects how ChatGPT surfaces content.
The encouraging part, and one that reinforces the on-site navigation use case, is that several of these providers publish and reference llms.txt files within their own developer and agent documentation. OpenAI and Anthropic both maintain them, and Anthropic points to the approach in its guidance on writing for agents. That signals real utility for helping agents work within a site.
For teams, the takeaway is reassuring: the consensus is remarkably aligned, which removes the guesswork. You can implement llms.txt as a clean, forward-looking measure for on-site agent navigation today, working in line with where the industry is heading rather than betting on one platform’s roadmap.
So does llms.txt have value? Yes, and here is where
The good news is that llms.txt has a clear and defensible role once you frame it correctly, with its core strength helping an AI agent that is already on your site to navigate efficiently, using minimal tokens to reach your most valuable content. That is a real benefit, and it becomes more relevant with every step towards agent-led browsing.
Several situations stand out as strong candidates.
Documentation-heavy and developer-focused sites gain the most, because a curated map to current, authoritative resources is genuinely helpful to any agent working through technical material. Brands that care about accuracy and control benefit too, since the file lets you steer agents towards your preferred, up-to-date pages. There is also a sensible future-proofing argument: the file is small, quick to create, and carries no technical risk, so implementing it now positions you well should the standard mature.
For agentic and ecommerce use in particular, it is worth keeping an eye on WebMCP, which focuses on giving agents actionable capabilities such as filtering products, comparing options, and completing tasks. llms.txt and WebMCP are complementary parts of the same broader movement towards machine-friendly web experiences, and both reward teams who think ahead.
Should you add llms.txt to your site?
My consultative view, separating durable practice from passing hype, is that llms.txt is a reasonable and low-risk addition for the right sites, provided your expectations are set with precision.
If you run a documentation-led, developer-facing, or content-rich site, I would happily recommend adding it. It is quick to produce, simple to maintain, and it prepares you well for an increasingly agent-driven web. Treat it as a helpful complement to your foundations, which continue to be well-structured HTML, robust structured data, and the traditional SEO signals that still govern discovery and ranking.
Approached this way, llms.txt is a small, sensible investment that supports the direction of travel while your core strategy does the heavy lifting.
How to create an llms.txt file, step by step
Getting started is genuinely quick, which is one of the file’s most appealing qualities.
Begin by auditing and curating your content. Decide which pages, documents, and resources best represent your site and would be most useful to an agent exploring it. Quality and relevance matter far more than volume here.
Next, create the file in Markdown (.md), following the clean structure described earlier: a clear heading, a short summary, and grouped links with brief descriptions. Keep the language plain and the selection focused.
Finally, upload the file to your domain root so it is accessible at yoursite.com/llms.txt. If you would rather automate the process, popular SEO plugins including Yoast and All in One SEO now offer llms.txt generation, which makes ongoing maintenance effortless (as with your sitemap.xml file).
Frequently Asked Questions: llms.txt
What is an llms.txt file?
It is a proposed standard: a Markdown file at your site root that offers AI systems a curated summary and a list of your most important content, designed to help agents navigate your site efficiently.
Does llms.txt actually work?
For its intended purpose of on-site agent navigation, yes. Current research shows it has no measurable effect on AI discovery or citation frequency, which aligns with its original design.
How do you make an llms.txt file?
Curate your key content, write it up in Markdown with a heading, a summary, and grouped links, and upload it to your domain root. Plugins such as Yoast and All in One SEO can generate it for you.
Is llms.txt the same as robots.txt?
They share a location and both address automated visitors, though their jobs differ. Robots.txt manages crawler access, while llms.txt guides and informs agents about your content.
