Broadnet/gemma-4-26B-A4B-uno-pilot-adapter

Uno pilot adapter: Gemma 4 26B A4B (pilot corpus, step 1900)

A rank-16 Uno adapter for `cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit`, measured at 1.1662x plain decoding on the 72-prompt production workload. It attaches to the Gemma 4 26B A4B mixture-of-experts model and is served with the Uno speculative-decoding method through the vLLM-Uno fork v0.3.0 or later. It supports the AWQ-4bit checkpoint pinned in the identity table below: that is the checkpoint it was trained on, and the one every number below was measured on. It is a pilot adapter trained on 965 rows, and the measured numbers are its own. It was trained and served on the same AWQ 4-bit checkpoint, on one 24 GB card; no full-precision weights were used at any point.

Links. Serving code and container: the vLLM-Uno v0.3.0 release (repository, image ghcr.io/brntech/vllm-uno:0.3.0), and the upstream pull request vllm-project/vllm#55947. Paper with every measurement behind this card: <https://doi.org/10.5281/zenodo.22820511>.

What it is

What it measures

The throughput, acceptance, cycle-timing and sampled-gate measurements below used the historical Gemma overlay on the first-generation runner, with a paired plain control in the same serving session on an NVIDIA RTX 3090 under vLLM 0.28.1rc1.dev437 (seed 29). The v0.3.0 release profile, on the second-generation runner, is the one certified and launched below. The throughput rows use the 72-prompt production-sampling workload; cycle timing uses five fixed 384-token greedy requests after two warm-ups.

The gain over the earlier 400-update point of the same lineage is concentrated in the deep draft positions (position 4: 0.0502 → 0.1251 survival), not position 1. The lossless result reads "no discrepancy detected at this sample size", not proven equivalence, the gate's own qualification.

What it is not

• —Pilot scope. 965 training rows, a corpus whose measured ceiling is ~225 accepted of the 649 fixed-reference prefix candidates, one seed, one serving box. The measured numbers are its own.
• —Not a general quality upgrade. The adapter is for throughput, not quality.
• —Not "dense-equivalent cost". The measured cycle cost rises with draft width (real C(4) ≈ 2.53); at K=4 the forced-acceptance ladder is close, at K=8 it is optimistic by 19%. Any speedup computed from a forced ladder above K=4 is an upper bound, not an estimate.
• —Pilot corpus only. 965 rows from four public datasets; a larger corpus from the production recipe will replace this pilot.

Compatibility

The adapter is a rank 16 LoRA (alpha 256) on the query, key, value, output, gate, up and down projections of every layer of google/gemma-4-26B-A4B-it. This adapter was trained and evaluated on the pinned AWQ checkpoint named above; compatibility, acceptance, and throughput on other checkpoints require validation. On the tested card in the v0.3.0 release, lossless is used in this sense: Uno is as close to plain as plain is to itself across sessions on this hardware, under greedy and sampled decoding alike. It does not apply to other Gemma 4 sizes (the 31B dense or E4B models have different shapes).

How to run it

The adapter is loaded by the vLLM-Uno fork, v0.3.0 or later. The serving conventions below are the ones the adapter was verified under on the fork's Gemma 4 port (the release's docs/validation.md records the runs; the refusals such as language-only and single-KV-group requirements are enforced by the fork):

• —--dtype bfloat16, --attention-backend TRITON_ATTN (required: the decoder layers mix sliding-window and full attention, and on this build the FlashInfer path's one-plan window assertion kills Uno on the first request. FlashInfer's supported head sizes cover this checkpoint, so the blocking failure is the mixed layer configuration, not head size, and every published Gemma Uno number is a Triton-attention number)
• —--language-model-only, --disable-hybrid-kv-cache-manager
• —--max-model-len 8192, --max-num-seqs 4, --max-num-batched-tokens 2048, --gpu-memory-utilization 0.85
• —Uno K=4 with the adapter: "num_speculative_tokens": 4, "uno_lora_path": "<adapter dir>", --max-lora-rank 16
• —Draft MoE top-k 4 (UNO_DRAFT_MOE_TOPK=4) is part of the measured serving profile: the measured run's engagement line records moe_top_k=4, and the variant rejects an uncaptured serving shape by name under traffic rather than falling back to eager routing. --enforce-eager is a diagnostic flag, not a serving suggestion.
• —The release profile carries two LoRA slots (--max-loras 2, --max-cpu-loras 2), prefix caching on, pinned seed 29 and a fixed CUDA-graph capture list; the served model name is uno-gemma4-26b-a4b.

env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_LORA_ENABLE_DUAL_STREAM=1 python3 -m vllm.entrypoints.cli.main serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit   --served-model-name uno-gemma4-26b-a4b --host 0.0.0.0 --port 8000   --tensor-parallel-size 1 --api-server-count 1 --async-scheduling --generation-config vllm   --dtype bfloat16 --attention-backend TRITON_ATTN --language-model-only   --disable-hybrid-kv-cache-manager --enable-prefix-caching --seed 29   --max-model-len 8192 --max-num-seqs 4 --max-num-batched-tokens 2048   --gpu-memory-utilization 0.85   --enable-lora --lora-dtype bfloat16 --max-lora-rank 16 --max-loras 2 --max-cpu-loras 2   --lora-target-modules qkv_proj o_proj gate_up_proj down_proj   --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,5,6,7,8,13,14,15,16]}'   --speculative-config '{"method":"uno","uno_lora_path":"/path/to/adapter","uno_mask_token_id":262144,"uno_noise_seed":29,"num_speculative_tokens":4,"uno_noise_low":0}'

This is the command the release launcher's gemma4 profile builds (UNO_PROFILE=gemma4 bash release/serve.sh cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit <adapter directory>), including the Gemma-specific mask token, noise seed and noise floor in the speculative config. The draft MoE top-k variant is an opt-in (UNO_DRAFT_MOE_TOPK=4 in the environment) and is not part of this command. Run the launcher to reproduce the certified profile exactly rather than rebuilding the command by hand.

Notes:

• —The speedup row above was measured at --max-num-seqs 1 with the fixed request set; the port verification ran at --max-num-seqs 4. Both use the same Uno K=4 speculative config.
• —Serve the AWQ-4bit checkpoint the adapter was trained against; the adapter is a rank-16 LoRA, and the measurement launcher asserted the adapter's adapter_model.safetensors hash before starting each arm.
• —num_speculative_tokens 4 is the measured knee; K=8 is break-even and costs more per cycle.

Citation

If you use this adapter or the measurements above, cite the accompanying paper (BroadNet Research, 2026, https://doi.org/10.5281/zenodo.22820511):

@report{al_zubaidi_uno_moe_2026,
  title       = {Self-drafting with Uno on a mixture of experts: serving and training an adapter for Gemma 4 26B A4B},
  author      = {BroadNet Research Team},
  affiliation = {BroadNet},
  year        = {2026},
  doi         = {10.5281/zenodo.22820511},
  url         = {https://doi.org/10.5281/zenodo.22820511},
  note        = {Preprint}
}

The dense-target study, Uno in vLLM: An Independent Implementation and Empirical Serving Study (DOI 10.5281/zenodo.22652609), covers the Qwen3-8B results and is the implementation reference for that target; cite it for those results, not for this adapter's measurements.

Credits

Uno, the method this adapter implements, and the training recipe it follows are the work of Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing and Zhengzhong Liu: Unlocking Lossless Speedups in LLMs via Discrete Diffusion, arXiv:2609.04010, 2026 (<https://arxiv.org/abs/2609.04010>; code and adapters <https://github.com/ifm-ai/uno>), from the Institute of Foundation Models (IFM) and collaborators. Please cite their paper for the method.

Gemma 4 26B A4B and its multi-token-prediction assistant drafter are by Google; the AWQ 4-bit checkpoint this adapter attaches to is by cyankiwi (<https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit>); the DFlash drafter used as a comparator is by z-lab (<https://huggingface.co/z-lab/gemma-4-26B-A4B-it-DFlash>). vLLM is the serving engine this work extends.

This adapter, the Gemma 4 port and every measurement on this card are an independent implementation by the BroadNet Research Team.

Licence and where the training data comes from

The adapter weights are Apache 2.0, like the Gemma 4 base model they attach to. The quantized checkpoint you serve them with belongs to its authors.

The adapter was trained on 965 rows from four public datasets: DataCreatorAI/Multi-Turn-Conversational-SFT (Apache 2.0), nvidia/Nemotron-Post-Training-Dataset-v1 (CC BY 4.0, thanks to NVIDIA), m-a-p/CodeFeedback-Filtered-Instruction (Apache 2.0) and RASHID778/arabic-instruct-curated (Apache 2.0). None of the rows were written by the base model. One thing worth knowing before you build on it: 220 of the 965 rows (22.8%) come from the code-feedback set, whose own card says it contains text generated by OpenAI models. We kept those rows in this pilot and are saying so here so you can decide for yourself.