You ask ChatGPT which companies monitor AI search visibility, and it names three competitors. Not you. You check the basics: content is fresh, the sitemap is clean, robots.txt blocks nobody. The problem usually sits one step earlier. When an AI agent lands on your site for the first time, it reads navigation bars, cookie banners, and a wall of JavaScript instead of a short, honest description of what you do.
llms.txt exists for exactly that step. This guide turns the specification into actions: what the file is, how it divides work with robots.txt and sitemap.xml, how the four-layer structure works, eight mistakes that quietly waste the effort, and how to tell whether it did anything at all.
What llms.txt actually is
llms.txt is a Markdown file at a site's root, or at any subpath, that tells AI agents what the site is and where the important content lives, so an agent can find entry points without parsing the full HTML first.
Jeremy Howard of Answer.AI proposed it in September 2024; a v2 revision landed in 2026, documented at llmstxt.org. Be clear about its status: it is a proposal, not a W3C or search-engine standard, and no platform has publicly committed to reading every llms.txt it encounters. The ecosystem signals are still worth noting. OpenAI, Anthropic, and Google Gemini publish llms.txt files for their own developer documentation, and Chrome Lighthouse includes an llms.txt check in its agentic browsing audits.
The citable version: llms.txt does not control rankings. It lowers the cost of an AI system understanding your site, and lower cost raises the odds of being cited correctly.
How it divides work with robots.txt and sitemap.xml
The three files answer three different questions: robots.txt covers whether a bot may crawl, sitemap.xml covers which pages exist, and llms.txt covers which pages matter and how to read them. Conflating them weakens all three.
| File | Question it answers | Audience | Common misuse |
|---|---|---|---|
| robots.txt | Which paths may be crawled | All crawlers | Disallowing AI crawlers, which also removes any chance of citation |
| sitemap.xml | Every indexable page | Search engines | Renaming it to llms.txt |
| llms.txt | What the site is and where to start | AI agents and models | Marketing copy with no links or notes |
The practical difference is scope. A sitemap optimizes for completeness, often thousands of URLs that will never fit a context window. An llms.txt optimizes for selection, usually a few dozen links with a line of context each. The specification makes the division explicit: a sitemap is not a substitute for llms.txt. If the line between GEO and SEO is still unclear, read what GEO means first, then decide what belongs in the file.
The four layers of a valid file
A compliant llms.txt has four layers in a fixed order: an H1 with the site name, a blockquote summary, optional prose, and H2 sections holding link lists. You can skip layers, but the agent receives noticeably less.
| Layer | Syntax | Purpose | Required |
|---|---|---|---|
| 1 | # Site name | Declares who this is; the only required element | Required |
| 2 | > One-line summary | Tells an agent who you serve within seconds | Recommended |
| 3 | Plain paragraphs | Constraints, versions, definitions | Optional |
| 4 | ## Section plus [name](URL): note | A link list a parser can handle reliably | Recommended |
# AiVisible
> AI search visibility monitoring and GEO platform covering DeepSeek, ChatGPT, Perplexity, and Wenxin Yiyan.
## Core pages
- [Free scan](/research): Enter a brand, get visibility scores per platform
- [What is GEO](/blog/en/what-is-geo): Definitions, boundaries, and common misreadings
## Optional
- [Pricing](/#pricing): Plans and quotas
Two details get skipped most often. Links must be Markdown links, [name](URL), optionally followed by a colon and a note, because that is the form regexes and parsers handle consistently. And the conventional ## Optional section holds secondary material an agent may drop when context is tight.
The specification also recommends serving a Markdown version of key pages at the same URL, either as page.html.md or page.md, declared with rel="alternate" type="text/markdown". The llms.txt file itself can be referenced from pages with rel="describedby". This decides whether an agent following your links receives clean text or another HTML shell.
Five steps to ship one
Five actions take a site from nothing to a working file, roughly an hour once you have done it once. The hard part is not writing it; it is choosing which pages deserve a slot.
- Pick pages. List ten to twenty pages an agent genuinely needs: home, product or service, pricing, core concepts, FAQ, contact. Choose by the questions users ask, not by your navigation structure.
- Write the summary. One sentence in the blockquote covering who you are, who you serve, and what you cover. Keep the brand name and positioning identical to the rest of the site, since that consistency is what lets a model resolve your entity.
- Group. Use H2 sections by purpose, such as Core pages, Docs, and Case studies. Three to eight links per section; split anything larger.
- Add a note to every link. Ten to twenty characters or a short clause stating what the reader gets. An unannotated link gives an agent no way to rank it.
- Serve Markdown versions and test. Publish .md versions of key pages, then run the check the spec suggests: hand an agent nothing but your llms.txt and ask what the company does and which products it offers.
Eight mistakes that make it useless
These eight account for most files that exist but accomplish nothing. Auditing against the list is faster than rewriting.
- Links without notes. A bare list of URLs is a table of contents with no priorities.
- A renamed sitemap. Thousands of URLs never fit a context window, and there is no summary.
- Marketing copy. Phrases like industry-leading carry no information and dilute the facts around them.
- Summary that contradicts the site. Different names and positioning across pages prevent entity resolution.
- Links to HTML shells. Without a Markdown version, an agent parses your navigation and scripts again.
- Write once, never update. New product pages leave agents reading a stale map.
- Treating it as an indexing switch. It does not grant crawl access and does not guarantee a read; that is robots.txt territory.
- Using it instead of structured data. They complement each other: schema describes page entities, llms.txt describes site structure.
How to tell whether it worked
Three observable signals settle the question: how accurately an agent answers questions from your file, whether AI crawlers request it, and whether visibility scores move across platforms.
Start with a baseline. Run a free scan with AiVisible across DeepSeek, ChatGPT, Perplexity, and Wenxin Yiyan, recording scores and cited sources. Re-test every two weeks after shipping, watching specifically for your own pages appearing among the cited sources. Without a baseline, no change can be attributed to anything.
Then read the logs. Filter server access logs for GPTBot, ClaudeBot, and PerplexityBot, and check whether they requested /llms.txt and followed links into the .md versions. This is the hardest evidence available, and the step most teams skip.
Finally, run the agent test yourself. In a fresh session, provide only the llms.txt and ask three questions: what the company does, which products it offers, how to get in touch. Errors and omissions point directly at a weak summary or a badly grouped section. The specification suggests the same acceptance approach: let an agent judge whether the file works.
To place the file inside a broader process, see our walkthrough on running an AI visibility audit. llms.txt is one input to that process, not a standalone deliverable.
A real example to compare against
The easiest reference is our own: aivisible.top serves a minimal, spec-shaped /llms.txt that you can open and read line by line.
It opens with # AiVisible, follows with a blockquote summary naming the platforms it covers, then uses five H2 groups: core product, what GEO means, key pages, company, and contact. The whole file runs 31 lines. The key-pages section carries Markdown links with short notes, pointing at the homepage, the free scan entry point, and the Chinese and English blog indexes.
The deliberate tradeoff is what it leaves out. It links to the blog indexes rather than listing every published post, because the file's job is to route an agent to entry points, not to duplicate a site map. That distinction, entry points versus exhaustive inventory, is the clearest way to tell llms.txt from sitemap.xml.
You can restructure your own file the same way: get the H1 and summary right first, decide how many groups you need and how many links each holds, then annotate every link. Once the format is correct, the remaining work is editorial judgment.
When you can skip it
Not every site should invest here. Sites with fewer than ten pages, or businesses that close entirely on-site, should rank structured data and content quality ahead of llms.txt.
The reasoning is straightforward: the file reduces the cost of understanding a site. If the site holds nothing worth citing, cheaper comprehension changes little. Those sites gain more from rewriting each product page, filling in the FAQ, and shipping schema. llms.txt is an amplifier, not an engine.
The reverse also holds. If your content is already being cited but cited wrongly, with features misattributed or the company confused with a namesake, the marginal value is high, because the file addresses exactly that class of entity and structure error. Pair it with writing content AI systems can parse and the effect compounds.
FAQ
Is llms.txt an official standard?
No. Jeremy Howard and Answer.AI proposed it in 2024, and it remains a public proposal with the spec and discussion hosted at llmstxt.org and on GitHub. No search engine or AI platform requires it, though OpenAI, Anthropic, and Google Gemini publish llms.txt files for their own documentation.
Will adding llms.txt make AI recommend my brand?
No. It lowers the cost of understanding your site; it does not decide recommendations. Those depend on content quality, entity consistency, third-party mentions, and each platform's retrieval strategy. Treat it as foundational work rather than a switch.
Does llms.txt conflict with robots.txt?
They do not conflict because they answer different questions. robots.txt declares what may be crawled, while llms.txt provides a guided overview. Note that blocking user agents such as GPTBot or ClaudeBot in robots.txt also removes any chance of being cited by those platforms.
Do I need a Markdown version of every page?
No. Start with the pages listed in your llms.txt, declare them with the rel attribute, and extend coverage by traffic. Links that point to HTML-only pages force an agent to parse the page from scratch.
Can a site have more than one llms.txt file?
Yes. A file covers the pages under its path, so /docs/llms.txt covers everything in /docs/. When several files apply, agents should use the most specific one.