ec75hash/qwen36-arithmetic-readouts
Arithmetic Intermediate Readout Sensitivity on Qwen3.6-27B Arithmetic intermediate detection with a Jacobian lens depends sharply on the prompt token being read and the numeral forms accepted by the scorer. Across 105 order-of-operations items, the recorded hosted-lens responses contain the intermediate at rank 1 on 48 items at the trailing space, versus 4 at the preceding token. On the 25 held-out items with two-digit intermediates, rank-1 detection falls from 13 to 3 when… See the full description on the dataset page: https://huggingface.co/datasets/ec75hash/qwen36-arithmetic-readouts.
Arithmetic Intermediate Readout Sensitivity on Qwen3.6-27B
Arithmetic intermediate detection with a Jacobian lens depends sharply on the prompt token being read and the numeral forms accepted by the scorer. Across 105 order-of-operations items, the recorded hosted-lens responses contain the intermediate at rank 1 on 48 items at the trailing space, versus 4 at the preceding token. On the 25 held-out items with two-digit intermediates, rank-1 detection falls from 13 to 3 when Chinese numerals are removed from the synonym set. A separately fitted five-prompt lens also shows both sensitivities.
GitHub: code and report · Hugging Face: evidence and checkpoint · Full results · Methods · Provenance
Release v1.0.2: GitHub release · HF release snapshot · Reproduction CI
The package contains the original 55 evaluation items, 50 held-out items, 210 exact prompt-token sequences, 105 scored item rows, 1,349 raw evidence files, a CPU scorer, capture and fitting scripts, historical preregistration records, checksums, and the five-prompt lens checkpoint. The raw archive and checkpoint are hosted on Hugging Face; the ordinary GitHub clone stays small.
A hit means that an accepted single-token spelling of the intermediate reaches the specified rank at at least one evaluated layer, 0–62, at the final prompt token. For the five-prompt lens, layer 62 is a direct read of its target block; RESULTS.md also reports the fitted-source-only count. This is a vocabulary readout measurement. It does not establish which language the model computes in or that the intermediate is causally necessary.
The preregistered replication result is mixed: the correctly answered original items reach 16/47 (34.0%) at rank 1, below the registered 35% threshold. The strict sampled-continuation filter admits only 24/105 items, below the required 30, so that column cannot test the planned H1/H2 conclusions. Those outcomes are reported alongside the measurement sensitivities in RESULTS.md.
Reproduce the published numbers
Python 3.10+ is sufficient. This path uses the standard library and downloads an approximately 86 MB compressed archive; it requires no model, GPU, account, or API key.
git clone https://github.com/jeffreywilliamportfolio/qwen36-arithmetic-readouts.git
cd qwen36-arithmetic-readouts
python3 scripts/fetch_evidence.py
python3 analysis/score.py --check --out recomputedThe downloader pins the dataset revision in provenance/release.json, checks the archive SHA-256, and verifies every extracted file. The scorer independently aggregates those observations and compares every published item row and aggregate with the release tables.
To verify the preregistration freeze, run shasum -a 256 -c FREEZE.verify.sha256 (macOS) or sha256sum -c FREEZE.verify.sha256 (Linux) from the repository root: twelve OK lines confirm the published plan, addenda, runners, scorer, and item files are byte-identical to the locally recorded September 4 hash list. See `preregistration/README.md`. CI re-runs both checks on every push.
To also download the fitted lens, add --lens to the fetch command. It downloads approximately 3.25 GB. New model captures and fitting are described in METHODS.md; their numerical implementation follows the recorded pipeline, while their publication wrappers were checked with dry runs. No new GPU experiment was performed for this release.
Dataset layout and interpretation
The item_results viewer contains one row per item. group="paper" identifies the original 55; group="new" identifies the held-out 50. rank, no_space_rank, and no_cjk_rank use the hosted API's top-8 lists, with 99 meaning absent from the captured top 8, not a measured exact rank. hosted_n1000_rank and pile5_rank use local exact ranks. correct_* and admissible_* distinguish recorded flags from an independent continuation reparse. The prompt_tokens viewer contains the two prompt variants and their exact input IDs.
The held-out set was generated before its v2 captures, with seed 20260904, and deliberately requires at least one single-token route for each two-digit intermediate. Its performance should be interpreted for that reachable-item population. The release includes partial auxiliary API sampling records (74/105 items); the primary local sampling gate covers all 105 trailing-space prompts.
The original evaluation items and Jacobian-lens implementation come from anthropics/jacobian-lens at the recorded revision. The hosted checkpoint is distributed through neuronpedia/jacobian-lens. This release is Apache-2.0 within the scope described in NOTICE; calibration source material and external model dependencies retain their own rights.
Author: Jeffrey W. Shorthill. Captures: September 4–5, 2026. Offline audit and publication: September 14, 2026.