CoolFace
Datasetpublic

megrisdal/llms-txt

Context & Motivation https://llmstxt.org/ is a project from Answer.AI which proposes to "standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time." I've noticed many tool providers begin to offer /llms.txt files for their websites and documentation. This includes developer tools and platforms like Perplexity, Anthropic, Hugging Face, Vercel, and others. I've also come across https://directory.llmstxt.cloud/, a directory of… See the full description on the dataset page: https://huggingface.co/datasets/megrisdal/llms-txt.

sourceHugging Facecc0-1.0updated 3d agoView on Hugging Face
13likes113downloads
Dataset Card

Context & Motivation

https://llmstxt.org/ is a project from Answer.AI which proposes to "standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time."

I've noticed many tool providers begin to offer /llms.txt files for their websites and documentation. This includes developer tools and platforms like Perplexity, Anthropic, Hugging Face, Vercel, and others.

I've also come across https://directory.llmstxt.cloud/, a directory of websites that have /llms.txt files which is curated by these folks: https://x.com/llmsdottxt. I thought it would be fun to use this awesome resource to collect all of the files into a single dataset. They're simply markdown files. This dataset can then be used to build cool applications.

Thank you to Answer.AI and Jeremy Howard, the providers that are adopting this standard, and the maintainers of https://directory.llmstxt.cloud/.

How this dataset was made

This is the notebook that fetches files that linked to from https://directory.llmstxt.cloud/ and uses the kagglehub Python client library to publish the resulting output as this dataset.

Inspiration

  • —Give your LLM application access to this dataset to enhance its interactions with these tools, e.g., for code-generation tasks
  • —Search and knowledge retrieval
  • —Extract and summarize common developer tasks to generate novel benchmarks for LLM evaluation
  • —Validate the correctness of the llms.txt files

Contributing

I'd love if anyone is interested in contributing to improving the notebook that extracts the llms.txt files. Leave a comment on this dataset or on the notebook. Feel free to also ping me with interesting demos or applications you create with this dataset.

Photo by <a href="https://unsplash.com/@solenfeyissa?utmcontent=creditCopyText&utmmedium=referral&utmsource=unsplash">Solen Feyissa</a> on <a href="https://unsplash.com/photos/a-person-holding-a-cell-phone-in-their-hand-hWSNTPp4x4?utmcontent=creditCopyText&utmmedium=referral&utm_source=unsplash">Unsplash</a>