datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
llmstxt-corpus
The llms.txt corpus
Measurement data on the llms.txt convention, collected in one run on
5 August 2026.
llms.txt is a plain-text file at a site's root, proposed as a curated map
telling AI systems what the site contains. This is a measurement of what is
actually being published under that name.
Canonical release: https://doi.org/10.5281/zenodo.22859104
This repository mirrors that deposit. Cite the DOI, which always resolves to the
newest version.
Two observations… See the full description on the dataset page: https://huggingface.co/datasets/sempite/llmstxt-corpus.cleverhack-llms-txtDescription: cleverhack.com llms.txt file as a dataset
Last Update: 17 May 2026
llms-txt-directory-snapshot
llms.txt Directory Snapshot — May 2026
A snapshot of 735+ websites implementing the llms.txt specification as of late May 2026. Compiled and maintained by ThatDevPro as part of the Awesome-llms-txt community list.
What is llms.txt?
llms.txt is a proposed standard for websites to declare LLM-readable summaries at a stable URL. Crawlers (GPTBot, ClaudeBot, PerplexityBot) check /llms.txt and /llms-full.txt to retrieve a curated overview of the site without parsing the… See the full description on the dataset page: https://huggingface.co/datasets/ThatDeveloperGuy13/llms-txt-directory-snapshot.