abacusai/Smaug-Mini
818k
1---2license: apache-2.03library_name: transformers4base_model: Qwen/Qwen3.8-27B5base_model_relation: finetune6pipeline_tag: image-text-to-text7tags:8- agentic9- smaug10- abacusai11---12<div align="center">13 <picture>14 <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/63128dd099791aa61d180c72/dJ6uR23m09M-YEafBpmea.png" width="20%" alt="Smaug-Mini">15 </picture>16</div>17<hr>18<div align="center" style="line-height:1">19 <a href="https://abacus.ai" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Abacus.AI-white?color=1783ff&logoColor=white"/></a>20 <a href="https://huggingface.co/abacusai" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Abacus.AI-ffc107?color=ffc107&logoColor=white"/></a>21</div>22<div align="center" style="line-height: 1;">23 <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-f5de53?&color=f5de53"/></a>24</div>25 26## 1. Model Introduction27 28Smaug-Mini is an agentic finetune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), finetuned by Abacus.AI. It improves end-to-end agentic tool use and automation performance (+4.5 on AutomationBench, +17.1 on JobBench, +2.0 overall on LiveBench, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond at the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same `Qwen3_5ForConditionalGeneration` architecture with the vision tower intact, same 262,144-token context, same reasoning interface with `xhigh` / `medium` / `low` effort levels.29 30This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments.31 32## 2. Model Summary33 34<div align="center">35<table>36<tbody>37<tr><td align="center"><strong>Architecture</strong></td><td align="center">Dense hybrid-attention transformer + vision tower</td></tr>38<tr><td align="center"><strong>Total Parameters</strong></td><td align="center">27B</td></tr>39<tr><td align="center"><strong>Number of Layers</strong></td><td align="center">64 (48 linear-attention + 16 full-attention, 3:1 interleave)</td></tr>40<tr><td align="center"><strong>Attention Mechanism</strong></td><td align="center">Gated linear attention & full attention (GQA)</td></tr>41<tr><td align="center"><strong>Hidden Dimension</strong></td><td align="center">5120</td></tr>42<tr><td align="center"><strong>Number of Attention Heads</strong></td><td align="center">24 (4 KV heads)</td></tr>43<tr><td align="center"><strong>Vision Encoder</strong></td><td align="center">27-layer ViT, patch 16</td></tr>44<tr><td align="center"><strong>Vocabulary Size</strong></td><td align="center">~248K</td></tr>45<tr><td align="center"><strong>Context Length</strong></td><td align="center">262,144</td></tr>46<tr><td align="center"><strong>Multi-Token Prediction</strong></td><td align="center">1-layer MTP head (inherited; leave speculative decoding off)</td></tr>47<tr><td align="center"><strong>Precision</strong></td><td align="center">bfloat16</td></tr>48<tr><td align="center"><strong>Modality</strong></td><td align="center">Text, Image</td></tr>49<tr><td align="center"><strong>Base Model</strong></td><td align="center"><code>Qwen/Qwen3.8-27B</code></td></tr>50<tr><td align="center"><strong>Adaptation</strong></td><td align="center">On-policy RL (GRPO), LoRA merged as full delta (language trunk only)</td></tr>51</tbody>52</table>53</div>54 55## 3. Evaluation56 57### Smaug-Mini vs its base58 59*Higher is better. Base points from the Qwen3.8-27B model card and livebench.ai.*60 61<div style="max-width:1100px;margin:0 auto">62<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">63<thead><tr>64<th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th>65<th style="padding:10px 4px;text-align:center;font-weight:600;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;background:rgba(42,120,214,0.10);">Smaug-Mini</th>66<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.8-27B<div style="font-size:10px;font-weight:400;color:#6B6B6B">(base)</div></th>67<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.6-27B</th>68<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.7-Plus</th>69<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Opus4.6 Max</th>70</tr></thead>71<tbody>72<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Agentic</td></tr>73<tr>74<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">AutomationBench</td>75<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>41.8</strong></td>76<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.38)">37.3</td>77<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td>78<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">20.4</td>79<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">25.5</td>80</tr>81<tr>82<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">JobBench</td>83<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>50.5</strong></td>84<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">33.4</td>85<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">21.8</td>86<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.13)">27.6</td>87<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.27)">36.9</td>88</tr>89<tr>90<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench agentic coding</td>91<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.46)">60.8</td>92<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>61.4</strong></td>93<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">39.3</td>94<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td>95<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">49.0</td>96</tr>97<tr>98<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">NL2Repo-Bench</td>99<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>55.8</strong></td>100<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.18)">42.3</td>101<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">36.2</td>102<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">41.1</td>103<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.29)">47.6</td>104</tr>105<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Reasoning, knowledge & instruction following</td></tr>106<tr>107<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">GPQA-diamond</td>108<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.24)">89.4</td>109<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">89.2</td>110<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">87.8</td>111<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.35)">90.3</td>112<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>91.3</strong></td>113</tr>114<tr>115<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">HLE</td>116<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.32)">34.2</td>117<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">30.8</td>118<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">24.0</td>119<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.33)">34.7</td>120<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>40.0</strong></td>121</tr>122<tr>123<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">IFBench</td>124<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>82.0</strong></td>125<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">79.5</td>126<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.19)">69.1</td>127<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.41)">79.1</td>128<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">62.5</td>129</tr>130<tr>131<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench overall</td>132<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>76.9</strong></td>133<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">75.3</td>134<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">64.0</td>135<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td>136<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.39)">74.5</td>137</tr>138<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Vision</td></tr>139<tr>140<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">MMMU-Pro</td>141<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.10)">75.6</td>142<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.16)">76.3</td>143<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td>144<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>80</strong></td>145<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td>146</tr>147</tbody></table>148<div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card unless noted; the Opus 4.6 AutomationBench score is from our own harness run and its JobBench score is a reported number, not our run; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); — = not reported.</div>149</div>150 151### LiveBench category profile152 153*Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).*154 155<div align="center">156 <img src="figures/smaug_mini_livebench_categories.png" width="100%" alt="Smaug-Mini LiveBench category profile vs Qwen3.8-27B board entry">157</div>158 159<details>160<summary><b>Notes</b></summary>161 162All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound.163 164Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.165 166</details>167 168## 4. Training Approach169 170Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release.171 172One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.173 174## 5. Known Behaviors and Limitations175 176Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills.177 178## 6. Deployment179 180Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation:181 182```bash183vllm serve abacusai/Smaug-Mini \184 --max-model-len 262144 \185 --reasoning-parser qwen3 \186 --enable-auto-tool-choice --tool-call-parser qwen3_coder187```188 189Recommended sampling for agentic use: temperature 1.0, top_p 0.95, reasoning effort `xhigh` (the default). Recent vLLM versions return the chain-of-thought in the `reasoning` field of the message; when replaying conversation history, pass reasoning back under both `reasoning` and `reasoning_content` keys for portability across serving versions. Leave MTP-based speculative decoding off (§4).190 191## 7. License192 193Apache 2.0, inherited from Qwen3.8-27B.194 195## 8. Citation196 197```bibtex198@misc{abacusai2026smaugmini,199 title = {Smaug-Mini},200 author = {Abacus.AI},201 year = {2026},202 note = {Agentic fine-tune of Qwen/Qwen3.8-27B},203 url = {https://huggingface.co/abacusai/Smaug-Mini}204}205```206 207The Smaug line and the DPO-Positive method behind it:208 209```bibtex210@article{pal2024smaug,211 title={Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive},212 author={Pal, Arka and Karkhanis, Deep and Dooley, Samuel and213 Roberts, Manley and Naidu, Siddartha and White, Colin},214 journal={arXiv preprint arXiv:2402.13228},215 year={2024}216}217```218 219## 9. Contact Us220 221If you have any questions, please reach out at [Abacus.AI](https://abacus.ai).222