llms.txt: what it is and how to create one (Next.js example)

Q: Do I need llms-full.txt?

The spec does not define an llms-full.txt. Some sites publish one with all their content concatenated, as their own convention.

llms.txt is a Markdown file served at /llms.txt that summarizes a site for language models and agents. It has a title, one sentence saying what the project is, and lists of links to the pages that matter, each with a short note. The proposal, by Jeremy Howard, defines the format in a few lines. It expects an agent to read or search the file to find what it needs, then follow the relevant links.

This post is hands-on only. You will write the file, generate it from a curated list, validate it with a script and publish it on a Next.js site, including what changed in v2. AI visibility strategy lives in the GEO pillar post.

Learn the format in five rules

The spec defines these parts, in this order:

An H1 with the name of the site or project. It is the only required part.
A blockquote with a short summary that holds the key information needed to understand the rest.
Zero or more blocks of any kind of text except headings, with details.
Zero or more H2 sections with "file lists".
Each list item is a required Markdown link, [name](url), optionally followed by : and a note.

# Site name

> One or two sentences saying what it is, who it is for and what is here.

Details that help read the rest. No headings.

## Section

- [Page](https://example.com/page): short note about what it contains

Two changes in v2, from August 2026 that older tutorials still miss:

The Optional section lost its special meaning. In v1 it told the llms_txt2ctx tool what to drop when building context. The tool is no longer part of the proposal. The section is still allowed, as a convention for secondary links.
Discovery now exists. A page can point to the llms.txt that describes it with rel="describedby", and to its Markdown version with rel="alternate" type="text/markdown", through a <link> in the HTML or an HTTP Link: header.

List the pages in a curated file

llms.txt is a selection. A file that lists everything does not help an agent pick anything. Start with a short list in a YAML file that someone reviews:

site: Capiva
summary: >-
  Boutique AI consultancy (Brazil, US and UK). Combines business strategy with
  technical execution: diagnosis, proof of concept with real data and harness
  engineering for agents in production.
details: |
  - Portuguese content under /pt/ and English content under /en/, with the same slugs.
  - For pricing and scope questions, use the service pages; the blog is educational.
sections:
  Services:
    - title: Strategic Diagnosis
      url: https://capiva.tech/en/strategic-diagnosis
      note: 3 to 8 week investigation that maps where AI creates value in the operation
    - title: Technical Validation
      url: https://capiva.tech/en/technical-validation
      note: proof of concept with real data before the larger investment
  Guides:
    - title: What is an AI Harness
      url: https://capiva.tech/en/blog/what-is-an-ai-harness
      note: pillar post of the harness engineering cluster
    - title: What is GEO / SEO for AI
      url: https://capiva.tech/en/blog/what-is-geo-seo-for-ai
      note: pillar post of the AI visibility cluster
  Optional:
    - title: The Professional Uselessness Epidemic
      url: https://capiva.tech/en/resources/professional-uselessness-epidemic
      note: interactive essay about AI and work

The note is what the agent reads to decide whether to follow the link. "Learn more" says nothing. "3 to 8 week investigation that maps where AI creates value" says something.

Generate the file

#!/usr/bin/env python3
"""Generate llms.txt from a curated YAML file.

Usage: python3 generate_llms_txt.py pages.yaml > public/llms.txt
"""

import sys

import yaml


def generate(cfg: dict) -> str:
    parts = [f"# {cfg['site']}", "", f"> {' '.join(cfg['summary'].split())}", ""]
    if cfg.get("details"):
        parts += [cfg["details"].strip(), ""]
    for section, pages in cfg["sections"].items():
        parts.append(f"## {section}")
        parts.append("")
        for p in pages:
            line = f"- [{p['title']}]({p['url']})"
            if p.get("note"):
                line += f": {p['note']}"
            parts.append(line)
        parts.append("")
    return "\n".join(parts)


if __name__ == "__main__":
    with open(sys.argv[1], encoding="utf-8") as f:
        sys.stdout.write(generate(yaml.safe_load(f)))

pip install pyyaml
python3 generate_llms_txt.py pages.yaml > public/llms.txt

Output:

# Capiva

> Boutique AI consultancy (Brazil, US and UK). Combines business strategy with technical execution: diagnosis, proof of concept with real data and harness engineering for agents in production.

- Portuguese content under /pt/ and English content under /en/, with the same slugs.
- For pricing and scope questions, use the service pages; the blog is educational.

## Services

- [Strategic Diagnosis](https://capiva.tech/en/strategic-diagnosis): 3 to 8 week investigation that maps where AI creates value in the operation
- [Technical Validation](https://capiva.tech/en/technical-validation): proof of concept with real data before the larger investment

## Guides

- [What is an AI Harness](https://capiva.tech/en/blog/what-is-an-ai-harness): pillar post of the harness engineering cluster
- [What is GEO / SEO for AI](https://capiva.tech/en/blog/what-is-geo-seo-for-ai): pillar post of the AI visibility cluster

## Optional

- [The Professional Uselessness Epidemic](https://capiva.tech/en/resources/professional-uselessness-epidemic): interactive essay about AI and work

Notice the list before the first H2. It is allowed, because rule 3 only forbids headings there. Inside an H2 section, every item has to be a link.

Validate before you publish

The format is simple enough to break without anyone noticing. Someone adds an item without a link, a page changes URL, the separator turns into a dash. This validator checks the structure and, optionally, the HTTP status of each link. It does not follow redirects, so you see the 301 instead of hiding it:

#!/usr/bin/env python3
"""Validate an llms.txt against the llmstxt.org format.

Usage: python3 validate_llms_txt.py FILE_OR_URL [--check-links]
"""

import re
import sys
import urllib.request

ITEM = re.compile(r"^- \[(?P<name>[^\]]+)\]\((?P<url>[^)\s]+)\)(?P<rest>.*)$")


def read(source: str) -> str:
    if source.startswith("http"):
        req = urllib.request.Request(source, headers={"User-Agent": "validate-llms-txt"})
        with urllib.request.urlopen(req, timeout=15) as r:
            return r.read().decode("utf-8-sig")
    with open(source, encoding="utf-8-sig") as f:  # utf-8-sig: the BOM is allowed
        return f.read()


def http_status(url: str, method: str = "HEAD") -> str:
    class NoRedirect(urllib.request.HTTPRedirectHandler):
        def redirect_request(self, *args, **kwargs):
            return None

    opener = urllib.request.build_opener(NoRedirect)
    req = urllib.request.Request(
        url, method=method, headers={"User-Agent": "validate-llms-txt"}
    )
    try:
        with opener.open(req, timeout=15) as r:
            return str(r.status)
    except urllib.error.HTTPError as e:
        if e.code == 405 and method == "HEAD":  # server that does not accept HEAD
            return http_status(url, "GET")
        return f"{e.code} -> {e.headers.get('Location', '')}".strip(" ->")
    except Exception as e:  # noqa: BLE001 (DNS, timeout, TLS)
        return f"failed ({e.__class__.__name__})"


def validate(
    text: str, check_links: bool = False
) -> tuple[list[str], list[str], list[str]]:
    errors, warnings, links = [], [], []
    lines = [li.rstrip() for li in text.splitlines()]
    content = [li for li in lines if li.strip()]
    if not content or not content[0].startswith("# "):
        errors.append(
            "the first line with content must be an H1 ('# Site name')"
        )
    if sum(1 for li in lines if li.startswith("# ")) > 1:
        errors.append("there is more than one H1")
    if len(content) < 2 or not content[1].startswith("> "):
        warnings.append(
            "no summary blockquote right after the H1 (optional, but recommended)"
        )

    section = None
    for n, li in enumerate(lines, 1):
        if re.match(r"^#{3,} ", li):
            warnings.append(f"line {n}: the format only uses H1 and H2 ({li[:40]!r})")
        elif li.startswith("## "):
            section = li[3:].strip()
        elif section and li.startswith("- "):
            m = ITEM.match(li)
            if not m:
                errors.append(f"line {n}: list item without a markdown link [name](url)")
                continue
            links.append(m["url"])
            rest = m["rest"]
            if rest and not rest.startswith(":"):
                found = rest[:3].strip()
                warnings.append(f"line {n}: notes go after ':' (found {found!r})")
    tokens = len(text) // 4
    if tokens > 10_000:
        warnings.append(
            f"~{tokens} tokens (chars/4): the file should fit easily in context"
        )
    if check_links:
        for url in links:
            s = http_status(url)
            if s != "200":
                warnings.append(f"link {url} returned {s}")
    return errors, warnings, links


if __name__ == "__main__":
    source = sys.argv[1]
    errors, warnings, links = validate(read(source), "--check-links" in sys.argv)
    for e in errors:
        print("ERROR  ", e)
    for w in warnings:
        print("WARNING", w)
    print(f"{len(links)} links, {len(errors)} errors, {len(warnings)} warnings")
    sys.exit(1 if errors else 0)

$ python3 validate_llms_txt.py public/llms.txt --check-links
5 links, 0 errors, 0 warnings

The validator also accepts a URL, so you can run it against production. It exits with code 1 when there is an error, which is enough for CI. The 10,000 token limit (estimated as characters/4) is my choice, not the spec's. Adjust it to your case.

Publish on Next.js

A static file in public/ is served at the root. public/llms.txt becomes https://your-site/llms.txt with no code.

To implement v2 discovery, add the Link header to the pages the file covers:

/** @type {import('next').NextConfig} */
const nextConfig = {
  async headers() {
    return [
      {
        // Each post points to the llms.txt that describes it (v2 link relation)
        source: '/en/blog/:slug',
        headers: [{ key: 'Link', value: '</llms.txt>; rel="describedby"' }],
      },
    ]
  },
}

export default nextConfig

I tested it on a Next.js 16.4 app with next build && next start:

$ curl -sI http://localhost:3124/en/blog/x | grep -i link
Link: </llms.txt>; rel="describedby"
$ curl -sI http://localhost:3124/llms.txt | grep -i content-type
Content-Type: text/plain; charset=UTF-8

text/plain is what Next.js serves for .txt, and the spec does not ask for another type. The other half of v2, .md versions of each page (page.md or page.html.md), requires generating Markdown from the content. That waits until the blog has that output.

If the site has areas with different owners, v2 also defines files in subpaths. A /docs/llms.txt covers the pages under /docs/, and the most specific file wins.

How we use it at Capiva

I ran the validator against Capiva's llms.txt before writing this post. Result: 35 links, 10 errors and 37 warnings. The errors were items without links inside H2 sections (Services, Founder, Contact), written as bold text. Most warnings were the separator: we used — instead of : before the notes. Two links did not return 200. One pointed to an old domain that returned 404, and /en/blog redirected with a 308. We fixed the file and ran it again with --check-links: 34 links, 0 errors, 0 warnings. On October 12, 2026, after we merged four posts into the pillar posts, the same check gave 30 links, 0 errors and 0 warnings. None of those errors stopped a model from reading the file. They show the drift that happens when nobody validates. A validator in CI flags it right away, and a person reading the file by eye misses it.

Lighthouse has an llms.txt audit. If the file does not exist (404), the audit is marked not applicable. If the server returns an error when fetching it, the page gets flagged.

Troubleshooting checklist

[ ] The file does not open at /llms.txt. In Next.js it has to live in public/, not in app/. Check that no middleware or rewrite captures the route.
[ ] The validator reports an item without a link. Inside an H2 section, every item is - [name](url). Loose text goes before the first H2.
[ ] Links redirect. Use the final URL in the file. Each redirect is one more request for the agent.
[ ] Broken accents. Save as UTF-8. The spec allows a BOM, and the validator accepts it.
[ ] The Link header does not show up. The source in headers() has to match the exact route. Test with curl -I in production, because a CDN can drop headers.
[ ] Huge file. If it passed a few thousand lines, you built a sitemap. Cut it down to the pages that answer the questions of people looking for you.
[ ] The agent does not find the new page. llms.txt is static. Generate it at build time from the YAML so it does not fall behind.

Frequently asked questions

Does llms.txt replace robots.txt or sitemap.xml?▼

No. robots.txt says what can be crawled, the sitemap lists everything for indexing, and llms.txt is an annotated selection for models to read.

Do I need llms-full.txt?▼

The spec does not define an llms-full.txt. Some sites publish one with all their content concatenated, as their own convention.

Who publishes llms.txt?▼

OpenAI and Anthropic do, for their documentation, for example. That shows adoption on docs sites. It does not show that search engines use the file for ranking.

Does it guarantee ChatGPT will cite my site?▼

No. The file helps agents navigate your content. To know whether you are being cited, measure it.

llms.txt: what it is and how to create, generate and validate one (Next.js example)