hugging2021/llms-txt-mcp

LLMs.txt Generator

Key Capabilities:

• —🤖 Automatically browse, research and highlight key information and generates an llms.txt for a package/project/API.
• —👨🏻‍🔧 Helper converter to generate llms-ctx.txt or llms-ctx-full.txt from a provided llms.txt

Demonstration Videos

Generate `llms.txt` via MCP Server:

Generate the llms.txt for Mistral Python SDK and save it to the llms-txt/ folder.


`llms.txt` to `llms-ctx.txt` Conversion Showcase:

Convert the @llms.txt to the corresponding llms-ctx.txt file and save it to the llms-ctx-txt/ folder.


Why llms.txt?

• —🚀 Richer context than traditional documentation
• —🌐 Universally applicable across domains
• —🤖 Enhances AI-assisted development and communication

Alternatives exist, but there are trade-offs:

• —Cursor offers a @docs documentation indexing, which serves a very similar purpose. However, storage is limited to your plan
• —llms.txt allows the developer to highlight key information and gotchas, links to critical dependencies which in certain cases may provide a boost to the context shared with the LLMs (vs a document chunk retrieval approach)

See: https://www.answer.ai/posts/2024-09-03-llmstxt.html

Quick Start

Prerequisites

• —Mistral AI API
• —Weights & Biases (optional)
• —Brave Search API

Setup

# Install dependencies
uv pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys

# Run the application
uv run --env-file=.env src/main.py

Possible Future Roadmap

• —Expand to non-coding domain contexts (See: https://llmstxt.org/domains.html )
• —Create a centralized llms.txt database

Misc notes

1. 1.MCP clients:
2. 2.Start MCP Inspector: npx @modelcontextprotocol/inspector
3. 3.Use Another Gradio app as MCP client
4. 4.Register MCP server in Cursor/VSCode
5. 5.Ways to connect to the MCP Server:
6. 6.Connect to MCP Server via: https://shern-llms-txt-mcp.hf.space/gradio_api/mcp/http/
7. 7.Note the trailing slash helps in some cases
8. 8.Must make Huggingface Space public
9. 9.Connect to MCP Server locally (dev): http://127.0.0.1:7860/gradio_api/mcp/http/
10. 10.Connect to MCP Server via Cursor: use /gradio_api/mcp/sse endpoint instead for now.