llms.txt

llms.txt is a markdown file served from your site root (/llms.txt) that gives LLMs and AI agents a curated map of your content. Proposed by Jeremy Howard in 2024, it fills a gap robots.txt was never designed for: robots.txt says what may be crawled, while llms.txt says what the site is and where the substance lives.

The format

A minimal, well-formed llms.txt:

# Site Name

> One-sentence description of what this site is.

## Docs
- [Quickstart](https://example.com/docs/quickstart): Get running in 60 seconds

## Key Pages
- [Pricing](https://example.com/pricing): Plans and limits

The convention that works in practice: keep llms.txt short — a map, not the territory — and serve llms-full.txt alongside it for agents that want the full text of key pages in one fetch.

What it is not

• Not a permission system. Blocking an AI crawler happens in robots.txt or your WAF, not here.
• Not a ranking factor anyone has confirmed. It is context delivery. An agent that reads a good llms.txt arrives at your pages already knowing what they will find.
• Not a sitemap replacement. Sitemaps list URLs for crawlers; llms.txt curates meaning for models.

How to check yours

Our site audit tool flags llms.txt presence and structure as part of its AI-readiness checks, alongside robots.txt AI-crawler rules and FAQ schema. The check is deliberately boring: file exists, loads fast, links resolve, descriptions present — because that is all an agent needs from it.

Related concepts

• Query Fan-Out — how engines decompose a prompt before retrieving your pages
• AI Readiness — the broader audit layer llms.txt belongs to
• GEO Score — llms.txt is one input among sixteen

FAQ

What is llms.txt?

llms.txt is a markdown-formatted file served at your site root that gives LLMs and AI agents a curated map of your content — what the site is, what matters, and where to find it. It was proposed by Jeremy Howard in September 2024 as an analog to robots.txt, but for AI consumption rather than crawl permission.

Do AI search engines actually read llms.txt?

Adoption is partial and undocumented by the major engines, but AI agents and third-party crawlers increasingly fetch it, and it costs nothing to serve. Treat it as progressive enhancement: robots.txt governs crawling, llms.txt guides understanding. Sites that serve one report AI agents arriving with better context.

How is llms.txt different from robots.txt?

robots.txt is a crawl-permission file read by all crawlers — it says what may be fetched. llms.txt is a content map read by LLM-based agents — it says what the site is and which URLs carry the substance. You need both; they answer different questions.

What is llms-full.txt?

A companion convention: llms.txt stays short with links, while llms-full.txt inlines the full content of key pages for agents that prefer one large document. Keeping both lets lightweight agents get the map and heavyweight agents get everything.