Modelpublic

abacusai/Smaug-Mini

sourceHugging Faceapache-2.0updated 12d agoView on Hugging Face

8likes16kdownloads

Model Card

1. Model Introduction

Smaug-Mini is an agentic finetune of Qwen3.8-27B, finetuned by Abacus.AI. It improves end-to-end agentic tool use and automation performance (+4.5 on AutomationBench, +17.1 on JobBench, +2.0 overall on LiveBench, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond at the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same Qwen3_5ForConditionalGeneration architecture with the vision tower intact, same 262,144-token context, same reasoning interface with xhigh / medium / low effort levels.

This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments.

2. Model Summary

<div align="center"> <table> <tbody> <tr><td align="center"><strong>Architecture</strong></td><td align="center">Dense hybrid-attention transformer + vision tower</td></tr> <tr><td align="center"><strong>Total Parameters</strong></td><td align="center">27B</td></tr> <tr><td align="center"><strong>Number of Layers</strong></td><td align="center">64 (48 linear-attention + 16 full-attention, 3:1 interleave)</td></tr> <tr><td align="center"><strong>Attention Mechanism</strong></td><td align="center">Gated linear attention & full attention (GQA)</td></tr> <tr><td align="center"><strong>Hidden Dimension</strong></td><td align="center">5120</td></tr> <tr><td align="center"><strong>Number of Attention Heads</strong></td><td align="center">24 (4 KV heads)</td></tr> <tr><td align="center"><strong>Vision Encoder</strong></td><td align="center">27-layer ViT, patch 16</td></tr> <tr><td align="center"><strong>Vocabulary Size</strong></td><td align="center">~248K</td></tr> <tr><td align="center"><strong>Context Length</strong></td><td align="center">262,144</td></tr> <tr><td align="center"><strong>Multi-Token Prediction</strong></td><td align="center">1-layer MTP head (inherited; leave speculative decoding off)</td></tr> <tr><td align="center"><strong>Precision</strong></td><td align="center">bfloat16</td></tr> <tr><td align="center"><strong>Modality</strong></td><td align="center">Text, Image</td></tr> <tr><td align="center"><strong>Base Model</strong></td><td align="center"><code>Qwen/Qwen3.8-27B</code></td></tr> <tr><td align="center"><strong>Adaptation</strong></td><td align="center">On-policy RL (GRPO), LoRA merged as full delta (language trunk only)</td></tr> </tbody> </table> </div>

3. Evaluation

Smaug-Mini vs its base

Higher is better. Base points from the Qwen3.8-27B model card and livebench.ai.

<div style="max-width:1100px;margin:0 auto"> <table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px"> <thead><tr> <th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th> <th style="padding:10px 4px;text-align:center;font-weight:600;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;background:rgba(42,120,214,0.10);">Smaug-Mini</th> <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.8-27B<div style="font-size:10px;font-weight:400;color:#6B6B6B">(base)</div></th> <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.6-27B</th> <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.7-Plus</th> <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Opus4.6 Max</th> </tr></thead> <tbody> <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Agentic</td></tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">AutomationBench</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>41.8</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.38)">37.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">20.4</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">25.5</td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">JobBench</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>50.5</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">33.4</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">21.8</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.13)">27.6</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.27)">36.9</td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench agentic coding</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.46)">60.8</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>61.4</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">39.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">49.0</td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">NL2Repo-Bench</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>55.8</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.18)">42.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">36.2</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">41.1</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.29)">47.6</td> </tr> <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Reasoning, knowledge & instruction following</td></tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">GPQA-diamond</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.24)">89.4</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">89.2</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">87.8</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.35)">90.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>91.3</strong></td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">HLE</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.32)">34.2</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">30.8</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">24.0</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.33)">34.7</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>40.0</strong></td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">IFBench</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>82.0</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">79.5</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.19)">69.1</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.41)">79.1</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">62.5</td> </tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench overall</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>76.9</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">75.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">64.0</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.39)">74.5</td> </tr> <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Vision</td></tr> <tr> <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">MMMU-Pro</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.10)">75.6</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.16)">76.3</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>80</strong></td> <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td> </tr> </tbody></table> <div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card unless noted; the Opus 4.6 AutomationBench score is from our own harness run and its JobBench score is a reported number, not our run; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); — = not reported.</div> </div>

LiveBench category profile

Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).

<details> <summary><b>Notes</b></summary>

All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort xhigh, with generation budgets large enough that no score is truncation-bound.

Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.

</details>

4. Training Approach

Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release.

One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.

5. Known Behaviors and Limitations

Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills.

6. Deployment

Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation:

bash

vllm serve abacusai/Smaug-Mini \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Recommended sampling for agentic use: temperature 1.0, topp 0.95, reasoning effort `xhigh` (the default). Recent vLLM versions return the chain-of-thought in the `reasoning` field of the message; when replaying conversation history, pass reasoning back under both `reasoning` and `reasoningcontent` keys for portability across serving versions. Leave MTP-based speculative decoding off (§4).

7. License

Apache 2.0, inherited from Qwen3.8-27B.

8. Citation

bibtex

@misc{abacusai2026smaugmini,
  title  = {Smaug-Mini},
  author = {Abacus.AI},
  year   = {2026},
  note   = {Agentic fine-tune of Qwen/Qwen3.8-27B},
  url    = {https://huggingface.co/abacusai/Smaug-Mini}
}

The Smaug line and the DPO-Positive method behind it:

bibtex

@article{pal2024smaug,
  title={Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive},
  author={Pal, Arka and Karkhanis, Deep and Dooley, Samuel and
          Roberts, Manley and Naidu, Siddartha and White, Colin},
  journal={arXiv preprint arXiv:2402.13228},
  year={2024}
}

9. Contact Us

If you have any questions, please reach out at Abacus.AI.