llms.txt
Datasets
All datasets matching “llms.txt”llms.txt
llms.txt files extracted from the Common Crawl corpus
This dataset contains llms.txt files extracted from the Common Crawl corpus.
Specifically, we extracted response records matching */llms.txt and */llms-full.txt with status 200 and MIME-type text/plain or text/markdown.
What is llms.txt?
The /llms.txt file: A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.
Our own analysis of this dataset is available… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/llms.txt.llms-txt
Context & Motivation
https://llmstxt.org/ is a project from Answer.AI which proposes to "standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time."
I've noticed many tool providers begin to offer /llms.txt files for their websites and documentation. This includes developer tools and platforms like Perplexity, Anthropic, Hugging Face, Vercel, and others.
I've also come across https://directory.llmstxt.cloud/, a directory of… See the full description on the dataset page: https://huggingface.co/datasets/megrisdal/llms-txt.llmstxt-corpus
The llms.txt corpus
Measurement data on the llms.txt convention, collected in one run on
5 August 2026.
llms.txt is a plain-text file at a site's root, proposed as a curated map
telling AI systems what the site contains. This is a measurement of what is
actually being published under that name.
Canonical release: https://doi.org/10.5281/zenodo.22859104
This repository mirrors that deposit. Cite the DOI, which always resolves to the
newest version.
Two observations… See the full description on the dataset page: https://huggingface.co/datasets/sempite/llmstxt-corpus.llms-txt
Dataset with /llms.txt
This dataset contains the llms.txt and llms-full.txt files obtained from directory.llmstxt.cloud.
The llms.txt defined at https://llmstxt.org/:
A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.
Some sites are starting to adopt this standard, and they are being tracked by directory.llmstxt.cloud,
so we can scrape and collect them in a dataset.
How it was obtained
By running… See the full description on the dataset page: https://huggingface.co/datasets/plaguss/llms-txt.cleverhack-llms-txtDescription: cleverhack.com llms.txt file as a dataset
Last Update: 17 May 2026
llms-txt-directory-snapshot
llms.txt Directory Snapshot — May 2026
A snapshot of 735+ websites implementing the llms.txt specification as of late May 2026. Compiled and maintained by ThatDevPro as part of the Awesome-llms-txt community list.
What is llms.txt?
llms.txt is a proposed standard for websites to declare LLM-readable summaries at a stable URL. Crawlers (GPTBot, ClaudeBot, PerplexityBot) check /llms.txt and /llms-full.txt to retrieve a curated overview of the site without parsing the… See the full description on the dataset page: https://huggingface.co/datasets/ThatDeveloperGuy13/llms-txt-directory-snapshot.