Datasetpublic

violetxi/harvey-eval-recall-qwen35-9b-notes70-notecondtraj30-100m-think

harvey-eval-recall-qwen35-9b-notes70-notecondtraj30-100m-think Complete closed-book recall evaluation: 7,933 probes. One dataset repository for Qwen3.5-9B, the 100M notes + note-conditioned trajectory mixture, and no KL regularization. The train split contains evaluation records. Each row is one scored probe; this split name follows the existing evaluation dataset layout. Model, data, and KL condition Evaluated model:… See the full description on the dataset page: https://huggingface.co/datasets/violetxi/harvey-eval-recall-qwen35-9b-notes70-notecondtraj30-100m-think.

sourceHugging Facemitupdated 4d agoView on Hugging Face

0likes93downloads

Dataset Card

harvey-eval-recall-qwen35-9b-notes70-notecondtraj30-100m-think

Complete closed-book recall evaluation: 7,933 probes. One dataset repository for Qwen3.5-9B, the 100M notes + note-conditioned trajectory mixture, and no KL regularization. The train split contains evaluation records. Each row is one scored probe; this split name follows the existing evaluation dataset layout.

Model, data, and KL condition

—Evaluated model: violetxi/qwen35-9b-harvey-v4-notes-conditioned-100m.
—Exact evaluated model revision: 0c295885100d6eba4f514752aa081c5b0c73fdec.
—Base model: Qwen/Qwen3.5-9B.
—Training mixture: Harvey v4, 70% stored-note labels and 30% assistant labels from note-conditioned trajectories; 100M nominal supervised-token scale.
—KL condition: no-kl; coefficient 0.
—Evaluation probes: violetxi/wmrl-v4-factstore-probes, revision 3e924cf285c387f7445c239bcdfe1572e8cc02a9.

Evaluation protocol

The existing recall-probe evaluation uses no document access. QA runs use thinking enabled, temperature 0, and a 1,024-token output ceiling. Scoring uses normalized answer matching, negative-query abstention, and multiple-choice statement log probabilities. There is no GPT judge and one evaluation per probe, as requested for recall probes. The four-sample requirement applies to the separate open-book evaluations.

Selection uses seed 0 and a limit of 1,500 probes per form before adding the covered-probe IDs. The union contains 7,933 unique probes. The holdout column preserves the source split flag; the union contains both covered and holdout probes.

Scores

Form	Probes	Metric	Rate
Forward QA	2515	Correct	0.3129
Reversed QA	1839	Correct	0.2001
Negative queries	1799	Correct abstention	0.7004
Multiple choice	1780	Raw logprob accuracy	0.6191
Multiple choice	1780	Length-normalized logprob accuracy	0.5466

Rates are fractions. Aggregate metrics match the saved per-probe outcomes. Metric columns that do not apply to a probe form are null.

Files and columns

—data/train-00000-of-00001.parquet: browsable table with every probe outcome and explicit model/data/KL identity.
—results.jsonl: the same table as JSONL.
—scores.json: all saved per-probe scores and aggregate metrics.
—summary.json: completion status, model revision, and summary metrics.
—provenance.json: source dataset revision, selection recipe, source hashes, and protocol.
—manifest.json: publication file hashes.

correct and abstained are QA indicators; raw and norm are multiple-choice correctness indicators. probe_id, form, holdout, tier, and oversample identify each source probe and its selection metadata. Remaining columns identify the evaluated model, training data, scale, KL condition, and probe dataset revision.

The original recall scorer saved scores and indicators rather than model response text. This release contains those saved outcomes. Free-text relation metadata is omitted. Raw partial open-book attempts are not included.