twistshan/realistic-niah-count-empirical-law
Realistic NIAH Count Empirical Law V3.1 This private dataset is the frozen stimulus bank for the preregistered V3.1 realistic needle-in-a-haystack counting experiment. Registered grid Passage lengths: 1K, 2K, 3K, 5K, 8K, 10K, 15K, and 20K canonical tokens. Needle counts: 1-10, 12, 15, 18, and 20. Paired seeds: 1234-1263 (30 seeds per cell). Total: 112 N x L cells and 3,360 unique stimuli. The stimulus bank is shared across all registered models and prompt modes.… See the full description on the dataset page: https://huggingface.co/datasets/twistshan/realistic-niah-count-empirical-law.
Realistic NIAH Count Empirical Law V3.1
This private dataset is the frozen stimulus bank for the preregistered V3.1 realistic needle-in-a-haystack counting experiment.
Registered grid
- Passage lengths: 1K, 2K, 3K, 5K, 8K, 10K, 15K, and 20K canonical tokens.
- Needle counts: 1-10, 12, 15, 18, and 20.
- Paired seeds: 1234-1263 (30 seeds per cell).
- Total: 112 N x L cells and 3,360 unique stimuli.
The stimulus bank is shared across all registered models and prompt modes. It does not contain model responses.
Integrity
audit_report.json records the independent frozen-grid audit. The published snapshot passed all checks: 3,360 rows, 112 cells, 30 rows per cell, no missing or damaged needles, no extra city-score pairs, and no checksum failures.
The SHA256 of stimuli.jsonl is:
afed18fe24d3c684b7f342a3c5cc119fe3bd4033487d25c97cbbe4fc21c0d159See manifest.json, contamination_audit.json, cell_counts.json, and SHA256SUMS for the registered design and machine-readable provenance.
Protocol
- Protocol version:
realistic_niah_v3_1 - Stimulus schema:
realistic_niah_master_v3_1 - Canonical tokenizer:
Qwen/Qwen3-8B - Tokenizer revision:
b968826d9c46dd6066d109eabc6255188de91218 - Insertion depth range: 5%-95% of the final passage
- Haystack mode: multi-file, without repeated source documents within a row
The corresponding implementation and preregistration are maintained in the Realistic CoT NIAH Count project. Absolute Windows paths retained in the manifest document the machine-local freeze environment and are not required to load stimuli.jsonl.