commoncrawl/llms.txt
llms.txt files extracted from the Common Crawl corpus This dataset contains llms.txt files extracted from the Common Crawl corpus. Specifically, we extracted response records matching */llms.txt and */llms-full.txt with status 200 and MIME-type text/plain or text/markdown. What is llms.txt? The /llms.txt file: A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time. Our own analysis of this dataset is available… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/llms.txt.
llms.txt files extracted from the Common Crawl corpus
This dataset contains llms.txt files extracted from the Common Crawl corpus. Specifically, we extracted response records matching */llms.txt and */llms-full.txt with status 200 and MIME-type text/plain or text/markdown.
What is llms.txt? The /llms.txt file: A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.
Our own analysis of this dataset is available as a Hugging Face space: 🤗 [commoncrawl/llms.txt-report](https://huggingface.co/spaces/commoncrawl/llms.txt-report)
Crawl archives:
- CC-MAIN-2026-30
This is an experimental dataset release and additional versions are currently not planned. Additional information can be found on the Common Crawl blog.