zhengjiyue/spark-x25-congruence-study

A one-unit change that removes every solution

This repository contains an original 32-problem linear-congruence corpus, 64 real Spark-X2.5-1.7B BF16 responses, and reproducible evaluation code. Each of 16 solvable problems is paired with an unsolvable problem whose right-hand side differs by one. Both thinking settings were run once under a 2,048-token cap.

Strict full-answer scores were 17/32 without thinking and 27/32 with thinking. These are results on a small constructed diagnostic, not a standard benchmark or a general estimate of mathematical capability. Four thinking outputs reached the cap; one additional non-thinking answer was mathematically correct but failed the predeclared final-line format.

• —Read the complete report, per-request results, all raw traces, and reasoning inspection.
• —study-evidence.zip (in the publication bundle) preserves the scripts, original folder structure, frozen protocol, inputs, outputs, token IDs, execution events, environment and hash checks. It contains no model weights.
• —cases.json is the frozen original corpus. Its expected field is a reference answer and was never passed to the model.
• —Original corpus, scripts and report: MIT. The downloaded model and official runtime retain their own licenses. Codex was used for experiment design, coding, orchestration, scoring/trace inspection and writing; the retained model outputs are actual Spark generations.

Reproduce on Apple silicon

The recorded environment used Python 3.12.13, macOS 26.5.1, Apple M5 Pro and 64 GiB unified memory. Runtime versions are pinned in requirements.txt; the public BF16 model data needs about 3.4 GB of disk space. No paid API is needed.

After extracting the evidence archive, enter its root directory:

python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
git clone https://github.com/XHToken/Spark-MLX-LLM.git vendor/Spark-MLX-LLM
git -C vendor/Spark-MLX-LLM checkout de2b4379fa1e2f2e1f99d84c83f0e008f651d86c
.venv/bin/python -m pip install --no-deps ./vendor/Spark-MLX-LLM
.venv/bin/python download_model.py
.venv/bin/python verify_scoring.py
.venv/bin/python analyze_results.py
.venv/bin/python audit_evidence.py

The last three commands validate and analyze the existing supplied evidence; they do not rerun inference. audit_evidence.py requires the downloaded tokenizer, but does not load model weights or perform inference. Its token reconstruction check matched all 64 retained output texts in this run.

To run a new experiment, preserve the supplied evidence directories first, then execute:

mv run recorded-run
mv smoke recorded-smoke
.venv/bin/python run_eval.py --smoke
.venv/bin/python run_eval.py
.venv/bin/python analyze_results.py
.venv/bin/python audit_evidence.py

The inference commands refuse to overwrite evidence. If a run fails, preserve and report it instead of silently retrying. build_cases.py refuses to overwrite the frozen corpus. The original model revision is 448e61eb392c00f2c403185c5b56d5e0665bfaab; both downloaded weight-file hashes were verified against official LFS metadata. Installed runtime sources were checked against all six Python files in the pinned official checkout.

The run emitted a generic Transformers model-type warning while loading its tokenizer. Recorded inference used spark_mlx_llm.model.Model, 28 layers and 1,707,657,216 BF16 parameters. No cross-engine numerical parity test is claimed. Timings are observations from the recorded device and environment.

Event: HER Hack-Astron #6. Account handles: Hugging Face zhengjiyue, GitHub ago3x.