megrisdal/llms-txt

Context & Motivation https://llmstxt.org/ is a project from Answer.AI which proposes to "standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time." I've noticed many tool providers begin to offer /llms.txt files for their websites and documentation. This includes developer tools and platforms like Perplexity, Anthropic, Hugging Face, Vercel, and others. I've also come across https://directory.llmstxt.cloud/, a directory of… See the full description on the dataset page: https://huggingface.co/datasets/megrisdal/llms-txt.