CoolFace
Datasetpublic

Mikhail/mini-jev-runs

mini-Jev run records: 27 900 schema-driven decisions with full candidate logits Every record is one decision a frozen Qwen/Qwen3-4B-Instruct-2507 made about one field of a JSON schema that arrived with the request. The field was turned into a lettered multiple-choice question (A = pay_bill, B = bill_balance, …), the model ran one forward pass, and the answer was read from its next-token logits over the option letters. No token was generated. The records keep what such a run… See the full description on the dataset page: https://huggingface.co/datasets/Mikhail/mini-jev-runs.

sourceHugging Facemitupdated 4d agoView on Hugging Face
0likes81downloads
Dataset Card

mini-Jev run records: 27 900 schema-driven decisions with full candidate logits

Every record is one decision a frozen Qwen/Qwen3-4B-Instruct-2507 made about one field of a JSON schema that arrived with the request. The field was turned into a lettered multiple-choice question (A = pay_bill, B = bill_balance, …), the model ran one forward pass, and the answer was read from its next-token logits over the option letters. No token was generated.

The records keep what such a run usually throws away: the candidate logits in fp32, the normalized scores, the gap between the two best options, how much of the whole next-token distribution sat on the option letters, the gold position, and the hashes of the prompt and of the schema.

Code, figures and the write-up that produced this: https://github.com/r-ms/mini-jev

What you can do with it without a GPU

  • —Calibration. p_cand against pred_pos == gold_pos: reliability diagrams, ECE, per-k and per-candidate_mass strata. The scores here are deliberately not calibrated and are never called probabilities of being right; measuring how far off they are is an open question.
  • —Abstention rules. gap is the logit distance between the two best letters. Pick a threshold, see what accuracy and coverage you get, compare with the out-of-scope split.
  • —Position effects. read_letters_rotated_options is the same questions with the options cyclically shifted; gold_pos and pred_pos let you measure any position prior directly.
  • —Reproduce our numbers. The report in the repository is recomputed from exactly these files.

Configs

configrowswhat it is
read_letters13 500the main grid: 450 CLINC150 texts × 3 field conditions × k ∈ {2, 4, 8, 15, 16}, distractors drawn within the domain (the hard case)
read_letters_shared_prefix13 500the same grid with the text prefilled once and its KV cache broadcast to the questions
read_letters_rotated_options600option order cyclically shifted, 50 texts, k ∈ {4, 8}
read_letters_out_of_scope10050 out-of-scope + 50 in-scope texts, k = 4 plus a "none of the above" option
free_text_control200the same prompts answered by unconstrained generation (≤ 8 tokens), to check that the model writes the letter we read
cost_by_length9 + 3wall time and prefill tokens for three ways of answering, at 32 / 128 / 512 (split b2) and 2048 (split b3) input tokens

Fields of a decision record

fieldmeaning
key, text_id, condition, field, k, k_eff, policyidentity of the question; condition encodes the grid cell, policy how distractors were drawn
options, gold, gold_posthe option strings in the order shown, the correct one, its position (0-based)
pred, pred_pos, tiethe answer read from the logits and whether the top two tied
cand_logits_bare, cand_logits_spacefp32 logits of the k letters, without and with a leading space, recomputed from the last hidden state
p_cand, p_mergedsoftmax over the candidate letters; p_merged merges the two tokenizations
gaplogit distance between the best and the second best letter — the confidence signal
candidate_mass, logZshare of the model's whole next-token distribution that sits on the k letters, and the log partition function
argmax_class, argmax_id, argmax_tokenwhat the unconstrained argmax over the whole vocabulary was: bare_candidate means the model was going to answer with an option letter anyway
pred_pos_native_bf16, native_matches_fp32the same decision taken from the model's own bf16 head, and whether it agrees with the fp32 recompute
prompt_tokens, pad_tokens, forward_passes, batch_id, batch_size, batch_wall_sec, elapsed_seccost and batching of that record
prompt_sha, prompt_template_sha, schema_sha, seed, run_mode, model, model_revision, code_revprovenance: every join in the study is made on prompt_sha, and a run refuses to resume if run_mode changed
prefix_tokens_shared, suffix_tokens, text_prefill_tokens_totalonly in read_letters_shared_prefix: how the request split into the shared prefix and the per-question tail
shiftonly in read_letters_rotated_options: the cyclic shift applied to the option order

free_text_control records instead carry the generated raw_response, generated_tokens and finish_reason; cost_by_length records carry wall times and prefill tokens for the per-field, shared-prefix and joint-JSON ways at one input length and field count.

Headline numbers from these files

the unconstrained argmax is an option letter13 600 / 13 600 questions
candidate mass on the k lettersmin 0.99999624, median 1.00000000
fp32 ties0
reading vs grammar-constrained JSON, intent fieldΔ −0.22 pp, 95 % CI [−1.44, +1.04], 6750 paired observations on 450 texts
the unconstrained model writes the letter we read199 / 200 prompt-identical controls
"none of the above" on out-of-scope texts41 / 50 reading, 47 / 50 generating
shared prefix vs plain readingsame accuracy (0.8484 vs 0.8483), 59 / 13 500 answers flip, all near ties

Quick start

python
from datasets import load_dataset

d = load_dataset("Mikhail/mini-jev-runs", "read_letters", split="train")
r = d[0]
print(r["options"], r["p_cand"], r["gap"], r["pred"], r["gold"])

# accuracy and a simple abstention rule
import statistics
acc = sum(x["pred"] == x["gold"] for x in d) / len(d)
kept = [x for x in d if x["gap"] > 5]
print(acc, len(kept) / len(d), sum(x["pred"] == x["gold"] for x in kept) / len(kept))

Provenance and limits

Model Qwen/Qwen3-4B-Instruct-2507 (revision cdbee75f), bf16, greedy, eager attention, transformers 4.57.6, xgrammar 0.2.7, one RTX 4090. Texts are CLINC150 (plus / test, pinned revision), 3 texts per intent for all 150 intents plus 50 out-of-scope texts; the sample manifest with its hashes is in data/clinc_sample_manifest.json.

One model, one English dataset of short utterances, one prompt form. The normalized scores rank the options and come with a confidence gap; they are not calibrated probabilities. Generation arms of the same study are not included here.

License: MIT for these records. CLINC150 is © its authors under CC BY 3.0.