thebajajra/whetstone-Qwen-1.7B-generations

whetstone-Qwen-1.7B-generations

Paired verbose and compact-register reasoning traces for 2,414 maths problems, with per-trace follow-ability scores.

Each row holds a problem, the long chain-of-thought Qwen3-1.7B produced for it, a compact-notation rewrite of that same reasoning, token counts for both, and the scores used to measure how followable the compact version is to Qwen3-1.7B.

11,174,460 original think tokens → 750,087 compressed (14.9×).

Selection

Every row passed four independent filters:

1. 1.Well-formed — the rollout parses cleanly into think and answer segments.
2. 2.Verifier-correct — the final answer matches ground truth under a deterministic checker (extraction from the answer segment only).
3. 3.On-register — the compact trace uses the intended compact notation rather than reverting to prose.
4. 4.Faithful to its source — an LLM judge compared the compact rewrite against the verbose original and confirmed the reasoning survived. The rubric permits dropping narration, self-talk and repetition; it forbids dropping any step's value, a case split, a rejected branch, or a self-correction.

The fourth filter is the restrictive one: of 2,674 problems eligible for it, 2,414 produced a trace that passed (92.3%), and the yield falls with difficulty — 98.9% at level 1, 65.6% at level 9.

Fields

The follow-ability scores

d_t is measured by teacher-forcing the compact trace through Qwen3-1.7B under the unprivileged prompt — the problem alone, with no answer key, no notation guide and no verbose trace. At each think token:

d_t = log p(most likely token) − log p(actual token)     ≥ 0

d_t = 0 means the model would have written exactly that token. Large d_t means a step it could not anticipate.

g_spike aggregates those gaps into a single follow-ability score:

g_spike = exp[ −(λ/β) · log( (1/T) Σ_t exp(β · d_t) ) ]     λ = 1

Higher `g_spike` = smaller gaps = easier for the model to follow. β controls how strongly the worst single gap dominates; both β=5 and β=10 are provided.

Caveat worth reading before using these as a quality signal

g_spike measures followability, which is not the same as correctness, and the two come apart as problems get harder. Measured against the judge's verdicts on 5,955 traces, its ability to separate faithful from unfaithful reasoning decays with difficulty:

At level 9 the underlying statistic inverts: faithful traces have higher mean d_t (0.824) than unfaithful ones (0.728). A genuine derivation of a hard problem contains steps a 1.7B model cannot anticipate — that is what makes the problem hard — whereas unfaithful reasoning tends toward generic, fluent, highly predictable filler. Treat g_spike as a usable weak signal on easy problems and as close to uninformative on hard ones.

Distribution

By source: 1,570 DeepMath, 844 GSM8K.

Level 1 is 35% of rows but only 15.2% of think tokens, while levels ≥6 are 56.3% of tokens. Sampling uniformly over rows and uniformly over tokens give noticeably different mixtures.

Compressed length is roughly flat with difficulty (126–402 tokens) while original length grows ~6× across levels, which is why the ratio falls from 8.7× at level 1 to 24× at level 9. On the hardest problems the compact trace is closer to a summary than to a re-encoding — the token budget stops scaling with the reasoning.

Sources

Problems come from GSM8K (level 1) and DeepMath-103K (levels 2–9), both under their original licences. The verbose traces are Qwen3-1.7B generations.