Yu-and-Ai/yutabase-reposearch-minieval

YUTABASE RepoSearch MiniEval

YUTABASE RepoSearch MiniEval is a tiny, project-specific retrieval check over one immutable public revision of `cambridgetcg/yutabase`. It asks 27 English, Cantonese Traditional Chinese, and code-mixed questions about the candidate specification, integration boundaries, optional SDK, and non-normative serving-shape research.

This is an engineering fixture, not a universal code-search benchmark. Its queries are synthetic and its public labels make memorization easy.

Configs

• —corpus contains exact, unmodified line spans from the allowlisted public source files. Its only split is also named corpus; it is not training data.
• —queries contains 9 validation and 18 test rows with graded relevance judgments and curated hard negatives. Both label sets are public.

from datasets import load_dataset

corpus = load_dataset(
    "Yu-and-Ai/yutabase-reposearch-minieval",
    "corpus",
    split="corpus",
)
validation = load_dataset(
    "Yu-and-Ai/yutabase-reposearch-minieval",
    "queries",
    split="validation",
)

Every corpus row carries the repository, commit, Git blob, whole-file SHA-256, inclusive line range, chunk SHA-256, authority class, and a commit-pinned source URL. source-manifest.json is the allowlist and reproducibility boundary.

Immutable source boundary

The corpus is generated only from 12 technical files named in the manifest. It excludes repository agent instructions, deployment metadata, heartbeat state, generated catalogs, applications, play notebooks, creative/archival writings, package locks, local repositories, account data, credentials, logs, and conversations. Links appearing in source documents are retained as source text; the linked pages are not copied into this dataset.

Authority classes matter. SPEC.md and docs/CONFORMANCE.md are candidate contract material, while docs/SERVING-SHAPE.md is explicitly non-normative research. Semantic similarity does not make those layers interchangeable.

Query and judgment shape

Each query row contains:

• —a stable query_id, split, language label, and query;
• —query_type and coarse difficulty;
• —an expected_facts audit aid;
• —graded relevance entries with a chunk ID, grade, and rationale; and
• —explicit hard_negative_chunk_ids.

Grades mean:

• —3: directly answers the information need;
• —2: materially corroborates or implements the answer;
• —1: supplies an important caveat or motivating constraint;
• —omitted: judged non-relevant inside this curated corpus.

The seed judgments were reviewed against the pinned corpus, but repeated facts across README, specification, code, and design notes can still make relevance subjective. Treat per-query output as evidence and inspect disagreements.

Language totals are 13 English (en), 11 Cantonese Traditional Chinese (yue-Hant), and 3 mixed (mul) queries. The source corpus itself is predominantly English, so the Cantonese rows measure cross-lingual retrieval, not Cantonese source coverage or answer generation.

Validate and evaluate

Everything below uses the Python standard library:

python3 scripts/validate_dataset.py
python3 -m unittest discover -s tests -v
python3 scripts/evaluate.py --split validation
python3 scripts/evaluate.py --split test

The evaluator's built-in lexical ranker is a deterministic smoke baseline, not a competitive multilingual model. A custom run is JSON Lines with one row per query:

{"query_id":"Q01","ranked_chunk_ids":["readme.profile","spec.profile"]}

python3 scripts/evaluate.py --split test --run path/to/run.jsonl

The primary metric is graded nDCG@10 with gain 2^grade - 1. Secondary binary metrics treat grades 2–3 as relevant: recall at 5, success at 1, and MRR at 10. A hard-negative inversion is 1 for a query when its highest-ranked curated hard negative appears above every grade-3 target, then the evaluator averages that value across queries. It also reports per-language macro scores and per-query results. With only 27 public questions, tiny score changes and language gaps are descriptive rather than statistically robust.

The checked-in standard-library lexical smoke baseline produced:

On the public test split its English nDCG@10 was 0.8818 and Cantonese Traditional nDCG@10 was 0.7726. That gap is a diagnostic for this lexical baseline, not an estimate of all multilingual retrievers or users.

Known limitations

• —The questions are manually authored from project documentation, not sampled from real users or production logs.
• —One repository and one revision cannot establish general retrieval quality.
• —The public test labels support regression testing, not a protected leaderboard.
• —Cantonese wording and code-switching cover only a small slice of Hong Kong usage and orthographic variation.
• —Hard negatives are curated rather than exhaustive.
• —A later YUTABASE revision requires a new dataset version; scores over different corpora are not directly comparable.
• —Retrieving a security statement does not prove a deployment secure, a claim true, or generated advice correct.
• —MIT permits redistribution of the selected source, but external material linked by that source retains its own terms and is not included here.

Privacy and safety

All source chunks come from the exact public Git commit above. No private AgentTool implementation, local KINGDOM catalog, local path, MCP/OAuth data, secret, user log, or conversation is included. The validator rejects several obvious local-path and credential markers as a regression guard; that check is not a universal secret scanner.

License and attribution

Dataset annotations and tooling are MIT licensed. The copied YUTABASE source spans remain under YUTABASE's MIT notice, reproduced in SOURCE_LICENSES/YUTABASE-MIT.txt.