commoncrawl/llms.txt

llms.txt files extracted from the Common Crawl corpus This dataset contains llms.txt files extracted from the Common Crawl corpus. Specifically, we extracted response records matching */llms.txt and */llms-full.txt with status 200 and MIME-type text/plain or text/markdown. What is llms.txt? The /llms.txt file: A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time. Our own analysis of this dataset is available… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/llms.txt.