dreeseaw/mdlens-realdocs-v1

mdlens-realdocs-v1

A held-out Markdown QA / retrieval eval built entirely from real open-source project documentation. It measures whether an agent can answer documentation questions from the right evidence with fewer irrelevant reads and fewer tokens.

Questions are deliberately low lexical overlap (paraphrased), so they stress retrieval rather than string matching. Every non-abstention question has its answer keywords verified to appear in the cited source file.

Question types

• —needle: a single fact stated in one section.
• —multi_hop: the answer requires combining facts from more than one place.
• —abstention: the answer is not present in the corpus; the correct behavior is to say the documentation does not specify it (this catches models that fabricate when retrieval returns near-but-irrelevant context).

Contents

Question schema: question_id, question, expected_answer, expected_answer_keywords, evidence_files, evidence_sections, type, difficulty, source_eval.

Sources, attribution, and licensing

This dataset redistributes documentation verbatim from the projects below, each under its own permissive license. The upstream license text is included under LICENSES/. All copyright remains with the respective projects.

Deliberately excluded to respect licensing: Redis (redis/docs is non-permissive / NOASSERTION), Prometheus (corpus was empty), and the Polars polars-cloud / polars-on-premises pages (commercial product docs).

If you are a maintainer of any source project and want a change in how your docs are represented here, please open an issue on the dataset.

Provenance of the questions

Questions were generated with an LLM over sections selected from the real corpus, then filtered: answer keywords must occur in the cited file, and phrasing is paraphrased away from the source wording. Abstention questions ask for facts confirmed absent from the corpus.

Intended use

Built to evaluate `mdlens`, a token-efficient Markdown retrieval CLI for agents, against plain shell retrieval (rg/cat). It is a Markdown QA/retrieval benchmark, not a general coding-agent benchmark.

Baseline results (v0.1.3, 28-question subset)

The runs in eval_results/ cover three harnesses and seven models: Claude Code (Opus 4.8, Sonnet 4.6), Codex (GPT-5.4, GPT-5.4-mini), and three open-source models via Pi on OpenRouter (Kimi K2.7, GLM 5.2, DeepSeek V4 Flash). Across all seven, the mdlens scout workflow beats plain shell retrieval: needle 75% to 86%, multi-hop 51% to 66%, average tool calls 7.8 to 4.9 per question, and lower cost on every harness that reports dollars (open-source models 33% to 41% cheaper). See eval_results/report.md for the full cache-aware breakdown.