violetxi/harvey-closed-book-qwen35-9b-notes-conditioned-30m-kl-0p05

harvey-closed-book-qwen35-9b-notes-conditioned-30m-kl-0p05

Completed closed-book C&H knowledge evaluation of violetxi/qwen35-9b-harvey-v4-notes-conditioned-30m-kl-0p05, revision 1bced92ee267198876b452d59e13cf17c41d5df9, job 122412. All 7,933 probes completed without API errors. Each probe category is a separate split in this single repo.

• —forward_qa: recover a fact value from its entity and relation.
• —reversed_qa: recover the entity from a fact value and relation.
• —mc_logprob: rank the true statement against three false statements by full raw sequence log-likelihood; norm is the separate length-normalized result.
• —negative_abstain: identify a perturbed false statement as false. Despite the historical name, this is a true/false test, not an unknown-answer rate.

No documents, search tools, or external judge are used. Chat probes enable thinking; their exact system prompt, token limits, and answer-matching rules are preserved in protocol/probe_score.py. Selection uses seed 0, 1,500 probes per category, plus the frozen covered-ID set, matching the existing baseline. covered denotes membership in that set; oversample is the saved flag for probes added beyond the initial random sample (null when not marked).

Saved-data limitation: the evaluation harness retained per-probe correctness and abstention flags, but discarded generated answer text, reasoning, candidate log-likelihood values, and predicted option indices. Those cannot be recovered from these files and have not been reconstructed. answer, label, and statement_true are reference targets, not model-generated outputs.

Parquet includes the original probe questions/statements, reference answers, metadata, model provenance, and recorded metrics. probe_json and result_json losslessly preserve each original source record. raw/results.json retains the entire saved evaluation, and summary.json recomputes exact means and counts. Mixed-type scalar probe fields use JSON text in their display columns; their original types are preserved in probe_json. The four splits are evaluation categories, not train/validation/test partitions. These probes measure knowledge in model weights and are distinct from the 250-task Harvey agentic benchmark and its all-criteria-pass score.