sulabhkatiyar/eagle3-sarvam-30b

Eagle3 Speculative-Decoding Draft Head for Sarvam-30B

A lightweight one-layer Llama-style Eagle3 draft head trained to accelerate inference for the 30B-parameter multilingual model `sarvamai/sarvam-30b` under speculative decoding. The draft proposes candidate continuations conditioned on auxiliary hidden states pulled from three mid-target layers; the target verifies the proposals in a single forward pass, accepting any agreed-upon prefix in one shot. This yields a 2.53x mean end-to-end speedup across MT-Bench, HumanEval, GSM8K and a multilingual prompt set, at zero quality cost (speculative decoding is mathematically lossless: the final output distribution is identical to the target running alone).

Eagle3 in one paragraph

Eagle3 (paper) trains a small auto-regressive draft head that ingests auxiliary hidden states from three layers of a frozen target model and emits proposed continuations one token at a time. At inference, the target verifies a tree of draft proposals in parallel and accepts the longest agreed-upon path, so each expensive target forward yields many emitted tokens instead of one. The draft is small (~4.7 GB here) and adds modest overhead per target step; net throughput rises in proportion to how many draft tokens the target ratifies.

Quick start (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="sarvamai/sarvam-30b",
    speculative_config={
        "method": "eagle3",
        "model": "sulabhkatiyar/eagle3-sarvam-30b",
        "num_speculative_tokens": 7,
    },
    trust_remote_code=True,
    dtype="bfloat16",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    disable_log_stats=False,  # required if you want real-time Mean Acceptance Length
)

prompts = ["Explain quantum computing in simple terms."]
out = llm.generate(prompts, SamplingParams(temperature=0.0, max_tokens=512))
print(out[0].outputs[0].text)

vLLM patches (required)

Stock vLLM does not load this draft out of the box. Five patches must be applied to your vLLM install. The patch script and its one supporting source file are bundled in this repo and can be downloaded and applied with three commands:

# 1. Download both files (the script + its sibling model file) into the layout it expects
mkdir -p vllm_patches scripts
curl -fsSL https://huggingface.co/sulabhkatiyar/eagle3-sarvam-30b/resolve/main/scripts/apply_vllm_patches.sh \
  -o scripts/apply_vllm_patches.sh
curl -fsSL https://huggingface.co/sulabhkatiyar/eagle3-sarvam-30b/resolve/main/vllm_patches/sarvam.py \
  -o vllm_patches/sarvam.py
chmod +x scripts/apply_vllm_patches.sh

# 2. Point the script at the venv where vLLM is installed (use $VIRTUAL_ENV after `source venv/bin/activate`,
#    or pass --venv /absolute/path explicitly). The script wants the venv root, not site-packages.
bash scripts/apply_vllm_patches.sh --venv "$(python -c 'import sys; print(sys.prefix)')"

The script is idempotent and self-verifies after every patch. Re-apply after any pip install / pip upgrade of vLLM (the patches are local edits to files inside site-packages/vllm/ and do not survive a vLLM reinstall).

A note on metrics: vLLM's LLM.get_metrics() returns 0 for speculative decoding unless disable_log_stats=False is passed to the LLM(...) constructor. The Mean Acceptance Length numbers in this card are scraped from the engine's SpecDecoding metrics: Mean acceptance length: X.XX log line, not from get_metrics().

Training data

The draft was trained on 247,959 multi-turn instruction-tuning records in SpecForge format: {"conversations": [{"role": "user", "content": ...}, {"role": "assistant", "content": ...}, ...]}. User prompts were sampled from public English and Indic corpora, then the assistant turns were re-generated by the target `sarvam-30b` itself with greedy decoding under vLLM so that the draft's training distribution matches what it will see at deployment time. About 94% of records have two turns (single-turn prompt/response) and the remaining ~6% have 4–20 turns (multi-turn instruction data).

Per-source counts (training mix)

The aggregate file was shuffled with seed 42 before writing, so the per-record language tag is not preserved inline. The breakdown above reflects the actual sampling-time arguments and matches the byte-for-byte counts on disk.

Roughly 0.2% (~484 / 247,959) of records were flagged by a heuristic degeneracy filter (repetition loops or truncation artifacts) but were kept. Filtering this category was found in prior experiments to regress draft acceptance — small amounts of imperfect responses make the draft more robust to non-canonical target output at inference time.

Training setup

Aux hidden-state convention (important for serving)

The draft consumes auxiliary hidden states from three of the target's 19 layers. SpecForge captured those during training via register_forward_hook, which delivers the output of layer i. The deployed config.json ships eagle_aux_hidden_state_layer_ids = [1, 10, 18] rather than the training-side [0, 9, 18]: vLLM's runtime captures the input to layer i (= the output of layer i-1), so the runtime indices [1, 10, 18] reproduce the same hidden-state set the draft was trained against. Two of three channels align perfectly; the third (layer 18) is the closest available approximation since the target has only 19 layers and there is no idx = 19. If you serve through a framework that captures layer outputs (matching SpecForge's training convention directly), revert the config to [0, 9, 18]. This is a single one-line change to the draft's config.json.

Results

All numbers below are from a single MI300X using vLLM with greedy decoding (temperature=0.0), num_speculative_tokens=7, max_tokens=512, max_model_len=4096, gpu_memory_utilization=0.9. Baselines are target-only with the same generation settings. MAL is the cumulative SpecDecoding metrics: Mean acceptance length line scraped from the vLLM engine log.

Per-dataset

[image]

[image]

Multilingual breakdown

The multilingual prompt set contains 50 free-form instruction prompts split as English 15 / Hindi 15 / Tamil 10 / Bengali 10 — i.e. 4 of the 12 languages the draft was trained on (see "Per-source counts" above for the full training mix). Per-language tok/s is averaged across that language's prompts; sample sizes are small, so per-language numbers are indicative rather than tight confidence intervals.

[image]

Limitations

• —Training mix is multilingual-heavy (Indic + English). Domains under-represented in the mix (e.g. very specialised technical content outside the math / code / reasoning slices) may see lower acceptance.
• —MAL plateaus near 3.3. The target's vocab is 262,144 and the draft is a single transformer layer; deeper or higher-capacity draft heads would likely lift this further. The draft was trained for one epoch and is at the natural plateau of that schedule.
• —Per-language eval set is small. English 15 / Hindi 15 / Tamil 10 / Bengali 10 prompts are enough for an indicative breakdown but not for tight confidence intervals.
• —Eight of the twelve trained languages have no measured speedup numbers. Telugu, Marathi, Kannada, Gujarati, Malayalam, Odia, Punjabi, and Urdu were in the training mix but were not in the evaluation set. Acceptance and speedup on those languages may be lower than English/Hindi due to smaller training volume per language and the absence of any direct measurement.
• —Aux-id calibration is framework-dependent. The shipped eagle_aux_hidden_state_layer_ids = [1, 10, 18] is correct for vLLM's input-capture convention. SGLang or any framework that captures layer outputs (matching SpecForge's training convention) should revert to [0, 9, 18].
• —License inherits from the base model. Use of this draft is bound by the licence of `sarvamai/sarvam-30b`.

Citation

If you use this draft, please cite both this artefact and the Eagle3 paper.

@misc{katiyar2026eagle3sarvam,
  title        = {Eagle3 Speculative-Decoding Draft Head for Sarvam-30B},
  author       = {Sulabh Katiyar},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/sulabhkatiyar/eagle3-sarvam-30b}}
}

@article{li2025eagle3,
  title   = {EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test},
  author  = {Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang},
  year    = {2025},
  journal = {arXiv preprint arXiv:2503.01840}
}

Acknowledgements

• —Sarvam AI for the sarvam-30b target model.
• —SpecForge for the Eagle3 training framework.
• —vLLM for the serving stack and the Eagle3 verification path.

License

Apache-2.0, matching the base model.