Wiself/Voice

1# Benchmarking methodology2 3Capability regression for base vs. voiced models. Standalone harness in4`benchmarking/`; only results and this methodology are published.5 6## Engine7 8[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)9(`lm-eval==0.4.13`, see `benchmarking/requirements.txt`) against a served10model over an OpenAI-compatible endpoint. No model weights are loaded by the11harness; inference stays in llama-server. Isolated venv, stdlib-only shim.12 13## Adapter: `benchmarking/shim.py`14 15llama-server and lm-eval disagree on three shapes; the shim translates,16nothing more:17 18- `GET /tokenizer_info` — missing on llama-server. Served from the server's19  own `/props` (bos/eos strings, canonical chat template).20- `POST /tokenize {"prompt"}` → upstream `{"content"}` → `{"tokens"}`.21- `POST /detokenize` — same translation, both directions.22- `/v1/completions` logprobs: server returns chat-style23  `content[{token, logprob, top_logprobs}]`; harness expects legacy24  `{token_logprobs[], top_logprobs[]}`. Translated field-for-field.25- Everything else is a transparent reverse proxy.26 27## Tokenizer28 29There is no local tokenizer file and none is downloaded. All tokenization is30performed by the served model itself via its own embedded tokenizer, which is31exact by construction. The chat template is the server's canonical template32for the served architecture (for Gemma 4 models: Google's canonical Gemma 433chat template). No gated downloads, no vocabulary mismatch possible.34 35## Tasks36 37| Task | Type | Items | Backend |38|---|---|---|---|39| tinyArc (logprob) | multiple-choice logprobs | 100 | ABANDONED, see below |40| tinyMMLU (logprob) | multiple-choice logprobs | 100 | ABANDONED, see below |41| arc_challenge_chat | generative letter answer | 100 (first of test) | `local-chat-completions`, server template |42| mmlu_generative | generative letter answer | 114 (2 × 57 subjects) | `local-chat-completions`, server template |43| tinyGSM8k | generation, exact match | 200 | `local-chat-completions`, server template |44 45Logprob scoring was abandoned: llama-server's `/v1/completions` does not46return prompt-token logprobs (`echo` unsupported), so multiple-choice47loglikelihood is unmeasurable on this endpoint. Generative letter-answer48scoring (the same style as OpenAI simple-evals MMLU) replaces it on every49task. Temperature 0.0 everywhere; greedy, deterministic.50 51## Results52 53Two files, temperature 0.0 throughout, same prompts, budgets, and54extraction on both:55 56- File A: `TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf`, unmodified.57- File B: `lm_head-TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf` — file A with58  its `lm_head` tensor replaced via `voice cast`, Q8_0. Nothing else changed.59 60| Task | n | A correct | B correct | A acc | B acc |61|---|---|---|---|---|---|62| ARC-Challenge (generative) | 100 | 94 | 95 | 0.940 | 0.950 |63| MMLU (generative, 2 × 57 subjects) | 114 | 101 | 103 | 0.886 | 0.904 |64| GSM8K | 100 | 76 | 92 | 0.760 | 0.920 |65 66Wrong/unanswered split — A: ARC 6/0, MMLU 5/8, GSM8K 1/23. B: ARC 5/0,67MMLU 7/4, GSM8K 2/6. Unanswered = empty model content after thought68exhausted the token budget (counted separately, never folded into wrong).69GSM8K sample files contain each doc twice (harness duplication); scores70dedup by doc_id, and split rescue pairs keep the decided verdict.71 72## Run labeling73 74Each run is labeled file-a or file-b in the model name and output path.75A run record consists of the harness `results_*.json` plus `samples_*.jsonl`76(every item) plus the shim's thinking log (every prompt, answer, and thinking77trace). Harness stock filters cannot read thinking-model output78formats (echoed answer prefixes, thought-first responses), so quoted scores79come from `benchmarking/rescore.py`, which extracts the final answer letter80(`answer is X` preferred, else last bare A-D) or number (`#### N`, else last81integer). Unanswered items (empty content after thought exhausted the token82budget) are reported separately, never silently folded into wrong.83 84Three thinking-model adaptations, all documented here so results stay85comparable: (1) system messages are folded into the user turn (Gemma86supports only user/model roles); (2) generation stop-sequences are87`["</s>"]` — task-default stops on `"\n"` decapitate thought mid-stream;88(3) token budgets 1024 (ARC/GSM8K) / 2048 (MMLU) to leave room for thought.89 90## Reproduce91 92uv venv benchmarking/.venv93uv pip install --python benchmarking/.venv/bin/python -r benchmarking/requirements.txt94python3 benchmarking/shim.py <llama-server-url> 808195benchmarking/.venv/bin/python -m lm_eval run --model local-chat-completions \96  --model_args base_url=http://127.0.0.1:8081/v1/chat/completions,model=<label>,num_concurrent=2,max_retries=3 \97  --tasks arc_challenge_chat --limit 100 --apply_chat_template \98  --gen_kwargs 'max_gen_toks=1024,until=["</s>"]' --log_samples --output_path benchmarking/runs/<label>-arc99benchmarking/.venv/bin/python -m lm_eval run --model local-chat-completions \100  --model_args base_url=http://127.0.0.1:8081/v1/chat/completions,model=<label>,num_concurrent=2,max_retries=3 \101  --tasks mmlu_generative --limit 2 --apply_chat_template \102  --gen_kwargs 'max_gen_toks=2048,until=["</s>"]' --log_samples --output_path benchmarking/runs/<label>-mmlu103benchmarking/.venv/bin/python -m lm_eval run --model local-chat-completions \104  --model_args base_url=http://127.0.0.1:8081/v1/chat/completions,model=<label>,num_concurrent=2,max_retries=3 \105  --tasks tinyGSM8k --apply_chat_template --gen_kwargs max_gen_toks=1024 \106  --log_samples --output_path benchmarking/runs/<label>-gsm8k107python3 benchmarking/rescore.py benchmarking/runs/<label>-arc/*/samples_*.jsonl108```109