Datasetpublic

sempite/llmstxt-corpus

The llms.txt corpus Measurement data on the llms.txt convention, collected in one run on 5 August 2026. llms.txt is a plain-text file at a site's root, proposed as a curated map telling AI systems what the site contains. This is a measurement of what is actually being published under that name. Canonical release: https://doi.org/10.5281/zenodo.22859104 This repository mirrors that deposit. Cite the DOI, which always resolves to the newest version. Two observations… See the full description on the dataset page: https://huggingface.co/datasets/sempite/llmstxt-corpus.

sourceHugging Facecc-by-4.0updated 2d agoView on Hugging Face

0likes48downloads

Dataset Card

The llms.txt corpus

Measurement data on the llms.txt convention, collected in one run on 5 August 2026.

llms.txt is a plain-text file at a site's root, proposed as a curated map telling AI systems what the site contains. This is a measurement of what is actually being published under that name.

Canonical release: https://doi.org/10.5281/zenodo.22859104 This repository mirrors that deposit. Cite the DOI, which always resolves to the newest version.

Two observations, six weeks apart

The identical 1,563 files were re-fetched on 20 September 2026 with the same code and user agent. llms.txt health got worse.

	2026-08-05	2026-09-20
Broken at file level	245 (15.7%)	265 (17.0%)

Healthy in both runs 1,280 · broken in both runs 227 · newly broken 38 · fixed 18.

227 files have been broken for at least six weeks. Files serving an HTML page instead of text nearly doubled, 32 to 60, while outright bot-blocking fell from 54 to 30. An HTML fallback returns 200 and looks healthy to anything that only checks a status code.

Link validation was not repeated, so every link-level figure below remains the 2026-08-05 observation.

What was measured

Ecosystem. All 1,563 llms.txt files registered in the convention's own public directories (llmstxt.site, directory.llmstxt.cloud, llmstxthub.com) were fetched, and 19,039 of the links inside the healthy ones were validated across 1,147 sites.

Shopify. 10,858 storefronts were checked for what they serve at /llms.txt: 8,951 custom-domain stores plus 1,907 canonical *.myshopify.com stores as a cross-check.

Findings

Measure	Value
Files broken at file level	15.7%
Files returning healthy text	84.3%
Files serving 403 to a research crawler	54
Links validated	19,039 across 1,147 sites
Links resolving cleanly	93.9%
Links dead	2.8%
Links blocked	2.1%
Sites linking to at least one dead page	18.2%
Healthy files containing no links at all	13.0%, about 1 in 8
Most links in a single file	11,137

Shopify storefronts. 7,133 of 10,858 returned a file. 6,016 of those are byte-identical: the file Shopify injects automatically. Among canonical *.myshopify.com stores it is 98%.

That file is not a content map. It is an Agent Instructions manifest describing how an agent should check out. Of the 7,133 files retrieved, 104 are a genuine curated content map, which is 1.5%.

File type	Share of 7,133
Shopify auto-file (Agent Instructions + Store Unavailable)	89.7%
HTML page, not a text file	3.9%
Misplaced robots.txt content	2.5%
Other or empty	2.4%
Genuine curated content map	1.5%

Configs

Config	Rows	What it is
`llmstxt_files`	1,563	one row per llms.txt fetched, with status and link count
`llmstxt_links`	19,039	one row per validated link, with its response status
`shopify_stores`	10,858	one row per storefront, with its sampling frame
`content_map_reclassification`	4	tally of the hand re-check, see limitations

Also in the repo: code/ with the collection and validation scripts, METHODS.md with the full limitations section, CHECKSUMS.txt.

How it was collected

Every fetch identified itself:

Mozilla/5.0 (compatible; SEMPITE-research/1.0; +https://sempite.com/research/)

A file counts as healthy when it returns HTTP 200 and the body is not HTML. An HTML body at /llms.txt is a fallback page, not an llms.txt file.

Links are checked HEAD first, GET on a HEAD-hostile response. Two tiers: every link up to a cap of 400 for roughly a hundred widely recognised documentation sites, and a seeded random sample of 15 links for every other site.

Limitations

Read METHODS.md for all eight. The four that matter most:

1.The ecosystem frame is self-selected. These are files someone registered in a public directory, skewed heavily toward developer documentation. This does not estimate llms.txt adoption across the web.

1.The Shopify frame is a sample, not a census. Public store lists over-represent active mid-market merchants and skew regionally.

1.Link health is a two-tier estimate. Long-tail sites contribute 15 sampled links each, so per-site figures for those carry real sampling error. The pooled rates are the defensible ones.

1.The Shopify file-type split rests on a recovered tally. The 825 stores first classified content_map were re-fetched and re-classified by a second pass that printed its result to a console rather than writing per-domain rows. The tally survived and ships as data/content_map_reclassification.csv, and the five percentages re-derive from it exactly, but you cannot look up which specific store was judged a genuine content map.

Every other figure re-derives from the raw rows here. The build script that produces this package recomputes all 25 published figures and refuses to package if any of them fails to match.

Citation

bibtex

@dataset{gerrish_llmstxt_corpus_2026,
  author    = {Gerrish, Jacob Ezra},
  title     = {The llms.txt corpus: ecosystem health, file contents,
               and the Shopify platform default},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22859104},
  url       = {https://doi.org/10.5281/zenodo.22859104}
}

Write-ups

—Ecosystem audit: https://sempite.com/research/llms-txt-audit/
—What is inside the files: https://sempite.com/research/inside-llms-txt/
—The Shopify default: https://sempite.com/research/shopify-llms-txt/

Data CC BY 4.0, code MIT. Produced by SEMPITE.