deepsweet/mdn

Pre‑ingested LanceDB of 50k+ rows from MDN Web Docs.

[!NOTE] Main intention is to use it with the companion RAG-MCP server ready for semantic search with hybrid vector (1024-d) and full‑text (BM25) retrieval. Other use cases are welcome as long as it complies with the license.

Content

The dataset covers the core MDN documentation sections, including:

• —Web API
• —JavaScript
• —HTML
• —CSS
• —SVG
• —HTTP

The source content, originally in a fairly custom markdown format, is thoroughly processed into plain yet still structured text and then semantically split into meaningful standalone chunks without a hard token limit or overlap.There may still be some minor nuances and room for improvement in the chunked text. Please let me know if you spot any quirks.

The embedding model used for data ingestion is BGE-M3 (1024 dimensions, 8192 tokens), specifically the quantized Q4_K_M GGUF version. Looking forward to trying pplx-embed-context-v1 once it's added to llama.cpp.

[!WARNING] It is strongly recommended to use exactly the same model for correct similarity during query embedding.

Usage

See development repo on GitHub for more details.

Articles

• —Парсим MDN и пишем оффлайн RAG-MCP (in Russian)

License

This dataset is definitely a "derivative work" based upon the original MDN Web Docs content which license is quite complex:

• —"Prose" content is available under CC-BY-SA-2.5.
• —Code examples and snippets added on or after August 20, 2010 are in the public domain (CC0).
• —Code examples and snippets added before August 20, 2010 are available under MIT.

All code examples are never explicitly modified during the chunking step, and therefore are provided as is.

This dataset as a whole is provided under the same terms. No new copyright is asserted.