llms.txt
A proposed convention for a Markdown file at the root of a website that gives AI systems a concise, curated guide to the site's most important content.
Definition
The llms.txt proposal, published in 2024 by Jeremy Howard, suggests that websites place a Markdown file at /llms.txt. The file starts with the site's name and a short summary, then lists the most important pages as links with one-line descriptions, grouped under headings. The aim is to give LLMs and agents, which have limited context and struggle with navigation, advertising and scripts, a clean map of what matters.
An optional /llms-full.txt can carry the full text of those pages in one document. The format is deliberately simple so that it can be generated from a sitemap or a build step. It is a proposal rather than a standard: some AI tools and documentation sites have adopted it, while the major search and answer engines have not confirmed that they use it. It does not control crawling; that remains the job of robots.txt.
Example: a developer-tools company publishes llms.txt listing its quick-start guide, API reference and pricing page with one-line summaries. A coding assistant fetching the file gets the right starting points in a few hundred tokens instead of crawling the whole site.
What it means in practice
- It is a low-cost way to point AI systems at your canonical, current content rather than whatever they find first.
- It forces a useful exercise: deciding which twenty pages define your business.
- Its influence on AI answer engines is unproven; it belongs inside a GEO plan and does not replace one.
- Generated automatically from your build, it stays current without extra effort.
How Teksmobile uses it
This website generates its own llms.txt at build time from page metadata, and our llms.txt and AI crawlers service does the same for clients, alongside crawler policy and structured data. It is one of the technical items in our generative engine optimization work.
Related terms
Part of the Teksmobile AI Glossary. See AI SEO and marketing.
Frequently asked questions
Do AI search engines actually read llms.txt?
Adoption is uneven and the major engines have not committed to it as a ranking input. It is cheap to publish, useful for developer tools and agents that fetch it directly, and harmless otherwise. Treat it as basic hygiene rather than a source of traffic.
What is the difference from robots.txt?
robots.txt tells crawlers what they may not fetch. llms.txt tells AI systems what is worth reading and where to find it. They complement each other.
What is llms-full.txt?
An optional companion file that contains the full text of the key pages in one Markdown document, so a model can load everything in a single request.