llms.txt, 90 Days Later: An Honest Retrospective

In June we shipped an llms.txt across aat.ee — one route file, a markdown map of the site's key content, served at aat.ee/llms.txt. It cost about an hour. Ninety days later, this is an honest account of what it did and did not do, because most of what you will read about llms.txt is either a spec rehash or a sales pitch, and the middle ground — measured results from a real production site — is thin.

What llms.txt is, in one paragraph

llms.txt is a proposed convention (not a standard — more on that later): a markdown file at the root of your domain that gives language models a curated map of your site. Where sitemap.xml is an exhaustive list of URLs for crawlers, llms.txt is a short, prioritised reading list for assistants — what matters, what it is called, and where it lives. The bet is that agents prefer a curated summary over crawling blindly.

What we shipped

The implementation is deliberately boring: a single dynamic route that composes three things — a static crawling-policy preamble, the 15 most recent launches, the 10 latest published articles, and the category list — into markdown, with one escape function wrapping anything user-controlled.

// User text (project names, article titles) must not break out of the
// markdown link or inject headings into an agent-facing file.
function mdSafe(text: string): string {
  return text
    .replace(/[\r\n]+/g, " ")
    .replace(/[[\]()]/g, "")
    .trim()
}

const projectLinks = recentProjects
  .map((p) => `- [${mdSafe(p.name)}](${baseUrl}/projects/${p.slug})`)
  .join("\n")

// User text (project names, article titles) must not break out of the
// markdown link or inject headings into an agent-facing file.
function mdSafe(text: string): string {
  return text
    .replace(/[\r\n]+/g, " ")
    .replace(/[[\]()]/g, "")
    .trim()
}

const projectLinks = recentProjects
  .map((p) => `- [${mdSafe(p.name)}](${baseUrl}/projects/${p.slug})`)
  .join("\n")

That sanitising step is the only subtle part. Project names and titles come from users and LLM pipelines; embedded newlines or brackets would let a listing inject a heading or a fake link into a file that assistants are invited to trust as a map.

What the logs showed

Three observations from ninety days of server logs and referrer data.

Agents do fetch it. We see periodic pulls of /llms.txt from known assistant crawler user-agents, at a much lower frequency than robots.txt but consistently present. Whatever the spec's status, the fetching behaviour is real.

No measurable referral miracle. AI-assistant referrals to aat.ee grew over the quarter — but the growth tracks our publishing cadence, not the llms.txt deployment. We cannot attribute a single session to an agent having read the file, and we are suspicious of anyone claiming tight attribution here: assistants fetch, then answer from a blend of sources, and the citation rarely carries a llms.txt-shaped referrer.

It forced a useful inventory. The unexpected benefit: writing a curated map means deciding what the top of your site actually is. Our route renders live data, so the file stays current — but the discipline of "what belongs in the reading list" improved the categories structure more than the file itself did.

The uncomfortable part: it is still a proposal

llms.txt has no ratified standard behind it. The MIME type is debated (text/markdown vs text/plain), assistants are not documented as reading it, and nothing breaks if you do not have one. Any post telling you it is "essential for AI SEO" in 2026 is ahead of the evidence.

Our position after ninety days: the cost is so low that the expected value stays positive, the way a well-organised lobby is positive for an office building. It will not drive the outcome by itself, but when an agent does go looking for orientation, you would rather it find a curated map than a wall.

Should you build one?

A decision table, honestly:

• If your site is under ~50 pages with clear titles — marginal. Your sitemap and navigation already tell the story.
• If you have a data-driven site (launches, listings, articles, docs — content that changes weekly) — yes. A generated llms.txt is one route file, stays current automatically, and gives assistants a spam-free summary that your sitemap cannot.
• If you were told it will boost AI rankings — no. It is a map, not a meta tag.

The implementation pattern above — compose, sanitise, serve — fits any framework with a routing layer. Keep it generated rather than hand-written, keep user text sanitised, and keep your expectations calibrated.