Schiltmans/bonsai2-drafter-eval

Bonsai 2 drafter evaluation corpora

Prompts and greedy responses from PrismML's Ternary-Bonsai-2-27B, recorded as token ids together with the per-round accepted lengths of the speculative decoding loop that produced them. The data exists to evaluate and train DFlash 2 drafters against this one target.

What it is for

• —Evaluating drafters. The eval rows of both files are the frozen inputs of the published benchmark protocol, BENCHMARK.md in github.com/alexschiltmans/bonsai2-drafter. general is the deciding suite there. code is reported alongside it but never decides, for the reason given under Limits.
• —Reproducing the published acceptance numbers. That repository's served-acceptance runner (bench/drafter/served_accept.py) feeds each eval row's prompt_ids to the served loop and writes a per-prompt report. The stdlib-only analyser (bench/analysis/analyse_served_accept.py) then computes paired acceptance from two such reports under the contract budgeted-prefix-identity/v2. It needs no model or GPU. Check the files against the sha256 values below before a run; the protocol identifies its inputs by them.
• —Training drafters. code.jsonl's train rows are the exact training corpus of `Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5`. Because decoding was greedy, every response is the target's own argmax path. round_lengths records where the served loop placed its anchors, which is what served-geometry training and served-anchored proxies need.

Files

general_chat.json is a JSON array of the 40 prompts behind general.jsonl, in the same order. Each entry has instruction, category and thinking. It is not one of the viewer's configs.

The JSONL files are byte-identical to the protocol's inputs. Nothing has been reformatted, re-sorted or cleaned since.

Loading

The Hub cannot split one file into several splits by a column, so the code config exposes all 300 rows as a single split named train_and_eval. Filter on the split field yourself:

from datasets import load_dataset

general = load_dataset("Schiltmans/bonsai2-drafter-eval", "general", split="eval")
code = load_dataset("Schiltmans/bonsai2-drafter-eval", "code", split="train_and_eval")
code_train = code.filter(lambda r: r["split"] == "train")  # 260 rows
code_eval = code.filter(lambda r: r["split"] == "eval")    # 40 rows

Decoding the token ids:

import json
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("prism-ml/Ternary-Bonsai-2-27B-mlx-2bit")

with open("general.jsonl") as f:
    row = json.loads(next(f))

print(tok.decode(row["prompt_ids"]))    # the chat-templated prompt the target saw
print(tok.decode(row["response_ids"]))  # the target's response; ends in <|im_end|> when finish == "stop"

Notes on the tokenizer:

• —A Qwen3.8-27B tokenizer decodes every row identically. This was checked on all 340 rows.
• —Use the recorded prompt_ids as they are. Re-applying the pack's chat template with its tokenizer, as in the snippet, reproduces all 340 of them. With thinking on, the template also inserts its default system message about reasoning effort. With thinking off, it closes an empty <think></think> block inside the prompt.
• —Recent transformers releases may warn on load about the tokenizer's regex pattern and suggest fix_mistral_regex=True. Do not pass that flag here. It changes the tokenization of 28 of the 340 prompts, so they no longer match prompt_ids. Decoding is unaffected either way.

Fields

How it was generated

Everything was produced by the repository's bench/drafter/gen_data.py, in process, with mlx-dspark's dflash_generate:

• —Target: prism-ml/Ternary-Bonsai-2-27B-mlx-2bit at revision 3f926b41, loaded through the repository's patched mlx-dspark 0.18.0 (MLX 0.32.2, mlx-lm 0.31.3) with an 8-bit KV cache.
• —Drafter: the stock z-lab/Qwen3.8-27B-DFlash2 (revision 50307d4c), quantized to 4 bits at load, which is mlx-dspark's default. Its block is 8 positions: the anchor plus 7 drafted tokens. The draft cap is 7, so every round verifies the full block.
• —Decoding: greedy (temperature 0). The loop verifies each drafted token against the target's argmax and keeps the longest matching prefix plus the target's own next token. The tokens are therefore the target's greedy output up to floating-point ties. The drafter changes how many rounds a response takes and never which tokens it contains. top_p, top_k and the seed are passed but have no effect at temperature 0.
• —Output budget: max_new_tokens of 1024 with thinking on and 400 with thinking off. The loop tests the budget before each round, not during it, so a length finish can overshoot by up to 7 tokens. Observed maxima are 1031 and 407. One code row that ends in stop is also over budget, because its end-of-turn token arrived in the round that crossed the budget.
• —`code` prompts: random.Random(7).sample(rows, 300) over code_alpaca_20k.json from sahil2801/CodeAlpaca-20k (revision 152bb5e9). Each prompt is the stripped instruction, followed by a blank line and the stripped input when there is one; an empty input or a <noinput> marker adds nothing. Rows are in sampled order. Row i (0-based) has thinking on unless i % 3 == 2, and rows 260 to 299 are eval.
• —`general` prompts: general_chat.json in file order, with each entry's own thinking and category. All 40 are eval.
• —Hardware: one Apple M4 Pro with 48 GB of unified memory.

The round_lengths rule

Anyone using these rows as served-geometry anchors needs this rule. It follows directly from mlx-dspark 0.18.0's dflash_generate.

1. 1.`response_ids[0]` belongs to no round. The loop seeds its output with the token the target picks from the prefill logits (out_ids = [pending]) before the first round runs.
2. 2.Round `r` is anchored at `response_ids[a_r]`, where `a_r = sum(round_lengths[:r])`. The round feeds the anchor and 7 mask slots to the drafter, verifies the 7 proposals, and commits round_lengths[r] tokens: the accepted proposals plus the target's own next token. So round_lengths[r] equals accepted + 1, between 1 and 8. Those tokens are response_ids[a_r + 1 : a_r + 1 + round_lengths[r]].
3. 3.The end-of-turn token is kept. A stop row's last token is <|im_end|> (id 248046). The loop also stops on <|endoftext|> (248044), but no row ends with it. A length row contains neither.
4. 4.The final round can be truncated, but its recorded length is not. When the end-of-turn token lands inside a round's committed block, the loop appends tokens up to and including it and discards the rest, while round_lengths still records the whole block. Only the last round can do this, because the loop ends there.

With k as the number of committed tokens discarded after the end-of-turn token:

len(responseids) == 1 + sum(roundlengths) - k

In this data k is never above 1. In general the final round emitted len(response_ids) - 1 - sum(round_lengths[:-1]) tokens, which lies between 1 and round_lengths[-1]. Every earlier round emitted its full recorded length. A round starts only while fewer tokens than the budget have been emitted, so the round that reaches the budget is always the last.

For training at the served anchors, take a_r = sum(round_lengths[:r]) for each round r. The drafter's 7 slots predict response_ids[a_r + 1 : a_r + 8], cut off at the end of the array. Those are the target's greedy tokens whether or not the stock drafter got them right. In a truncated final round, the slots past the end-of-turn token have no recorded target token.

The analyser in the GitHub repository applies the same arithmetic to its reports: a report's tokens is len(response_ids), which includes the seed token, and its rounds is len(round_lengths).

Intended uses

• —Measuring a DFlash 2 drafter's acceptance on Ternary-Bonsai-2-27B, greedy, against the published protocol and its analyser.
• —Training or fine-tuning a drafter for this target on code.jsonl's train rows, with round_lengths locating the anchors the served loop drew.
• —Checking that a runtime's greedy speculative loop is lossless. On the same runtime and settings, any drafter should reproduce these token paths within the budget.

Limits

• —One target, one runtime configuration. The responses belong to this pack of Ternary-Bonsai-2-27B at these settings, with an 8-bit KV cache. Another quantization, another KV precision or another runtime can take a different greedy path.
• —Greedy only. There are no sampled responses. Acceptance under sampling, including the target's published sampling defaults, has to be measured separately.
• —The `round_lengths` belong to the stock drafter. A different drafter commits the same tokens in different rounds.
• —Truncated responses are prefixes. length rows stop mid-answer. Of the thinking rows that hit the budget, 5 of 9 in general and 32 of 45 in code never close their reasoning block. Do not treat them as complete answers.
• —The `code` eval rows are not an untouched test set. They selected the training iteration of the ft5 drafter, so they are a regression screen. They are not an independent test of ft5 or of anything tuned on them. general.jsonl was not used for that selection, which is why the protocol decides on it.
• —Small. 40 eval prompts per suite, from one machine. The protocol reports paired bootstrap intervals for that reason.
• —Not a quality benchmark. The responses have not been checked for correctness. They record what the target said, not what it should have said.

Licence and attribution

This dataset is released under CC BY 4.0.

• —Code prompts: sampled from sahil2801/CodeAlpaca-20k by Sahil Chaudhary, CC BY 4.0. The prompts are reproduced verbatim apart from joining instruction and input.
• —General prompts: general_chat.json was written for this project.
• —Responses: generated by Ternary-Bonsai-2-27B (Apache-2.0), Prism ML's ternary model derived from Qwen3.8-27B. Created using Bonsai by Prism ML.

This is an independent project and not an official Prism ML, Qwen or z-lab release.

Links

• —Benchmark protocol, generator, runner and analyser: github.com/alexschiltmans/bonsai2-drafter
• —Drafter trained on code.jsonl's train rows: Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5
• —Stock drafter used for generation: z-lab/Qwen3.8-27B-DFlash2
• —Target: prism-ml/Ternary-Bonsai-2-27B-mlx-2bit

Citation

If you use the code prompts, please also cite Code Alpaca:

@misc{codealpaca,
  author = {Sahil Chaudhary},
  title = {Code Alpaca: An Instruction-following LLaMA model for code generation},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/sahil280114/codealpaca}}
}