tcclaviger/Tess-27B-RFA

1---2license: apache-2.03library_name: transformers4pipeline_tag: image-text-to-text5language:6- en7tags:8- quantized9- rfa10- 4-bit11base_model:12- migtissera/Tess-4-27B13base_model_relation: quantized14---15 16> [!IMPORTANT]17> ## All-RFA 4-bit quant of [migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)18>19> **Runtime:** requires [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) — an **RDNA 4 (gfx12xx)** vLLM image and the only build with the RFA kernels; no other vLLM build loads these weights. **Not validated on any other hardware at this time.**20 21# Tess-27B-RFA22 23Aggressive all-RFA 4-bit quantization of **Tess-4-27B** by Migel Tissera — an agentic,24thinking-native finetune of Qwen3.6-27B. Unlike the sibling25[Tess-27B-RFI](https://huggingface.co/tcclaviger/Tess-27B-RFI) (int8 attention),26this build runs **every attention path at 4-bit**: full attention, GDN linear27attention, and even the MTP speculative head's attention. All credit for the model28to its author; this repo only changes the numerics.29 30## Quantization by component31 32- **All attention (full self_attn + GDN linear attention) — 4-bit float weights (RFA)**: IQ4_NL non-linear grid, group size 16, asymmetric, Hadamard-16 rotation, block-float scales stored as int8 mantissa + int8 exponent.33- **MLP layers** — same RFA 4-bit scheme.34- **MTP speculative-decode head** — its attention projections are RFA 4-bit too; its fc and MLP stay bf16.35- **Kept in bf16 (not quantized)** — vision encoder, GDN `in_proj_a`/`in_proj_b`, embeddings, norms, and the lm_head.36 37Result: **20.5 GB** (vs 55.6 GB bf16, 28.9 GB RFI) at essentially zero measured quality cost.38 39## Serving context — 512K via YaRN40 41All evaluation below was run while serving at **`--max-model-len 524288`** (512K tokens),42extended from the native 256K window with YaRN via `--hf-overrides`:43 44```json45{"text_config": {"rope_parameters": {"rope_type": "yarn", "factor": 2.0,46 "original_max_position_embeddings": 262144, "mrope_interleaved": true,47 "mrope_section": [11, 11, 10], "partial_rotary_factor": 0.25,48 "rope_theta": 10000000}}}49```50 51## Evaluation results52 53Six builds measured with identical methodology, each against its own live vLLM endpoint (July 2026):54**Qwen3.6-27B** (bf16 base) → **Tess-4-27B** (the tune, bf16) → **Tess-27B-RFI** (int8+4-bit hybrid) →55**Tess-27B-RFA** (this build, all-4-bit) → **Tess-FP8** (W8A8 block-128 FP8 sibling),56with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference.57Bold marks the best score in each row (ties all bolded).58 59### Quantization cost (all-RFA 4-bit) — Tess-4-27B → Tess-27B-RFA60 61| Metric | Change |62|---|---|63| Checkpoint size | 55.6 → 20.5 GB (**−63%**) |64| WikiText-2 perplexity | 6.669 → 6.629 (**−0.6%, better**) |65| Codeneedle overall recall | 97.7% → 98.1% (**best of family**) |66| MC accuracy (4 tasks) | ≈ flat (−0.6 to +1.3 pp) |67| Tool-eval / GSM8K / MMLU / IFEval | 86 / 98% / 80% / 95% |68| MTP acceptance length | ~2.9 → ~2.8 (≈ flat) |69| Decode @ conc 1 (ISL 128) | 59.9 → 67.9 tok/s (+13%) |70| Decode @ conc 50 (ISL 128) | 528 → 423 tok/s (−20%) |71 72### Quality73 74| Metric | Qwen3.6-27B (base) | Tess-4-27B | Tess-27B-RFI | **Tess-27B-RFA** | Tess-FP8 | Qwen3.6-35B-A3B |75|---|---|---|---|---|---|---|76| WikiText-2 PPL (n_ctx 2048, lower is better) | 7.0559 | 6.6691 | 6.6632 | 6.6292 | 6.6627 | **6.5092** |77| ARC-Challenge (acc_norm) | 59.30% | **60.84%** | 60.41% | 60.32% | 60.49% | 55.20% |78| ARC-Easy (acc_norm) | 75.93% | 77.53% | 77.40% | **78.87%** | 77.82% | 71.13% |79| Winogrande (acc) | 77.51% | 77.43% | 77.51% | 76.80% | **77.66%** | 73.40% |80| HellaSwag (acc_norm) | 84.12% | 84.21% | **84.27%** | 84.05% | 84.13% | 82.95% |81 82Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.83 84### Long-context positional recall (codeneedle)85 86Verbatim function recall under 10K–80K-token contexts.87 88| Corpus | Qwen3.6-27B (base) | Tess-4-27B | Tess-27B-RFI | **Tess-27B-RFA** | Tess-FP8 | Qwen3.6-35B-A3B |89|---|---|---|---|---|---|---|90| Python | **100%** | **100%** | **100%** | **100%** | 99.55% | 99.09% |91| C++ | 98.12% | 98.12% | 98.44% | **98.75%** | **98.75%** | 98.44% |92| Rust | **99.69%** | **99.69%** | **99.69%** | **99.69%** | **99.69%** | 99.38% |93| JS (~80K tokens) | 93.44% | 93.13% | **93.75%** | **93.75%** | 93.44% | 92.19% |94| Tools | 98.26% | **99.57%** | **99.57%** | **99.57%** | **99.57%** | 93.48% |95| **Overall recall** | 97.81% | 97.73% | 97.97% | **98.05%** | 97.86% | 97.28% |96 97### Tool calling & accuracy benches98 99| Bench | Qwen3.6-27B (base) | Tess-4-27B | Tess-27B-RFI | **Tess-27B-RFA** | Tess-FP8 | Qwen3.6-35B-A3B |100|---|---|---|---|---|---|---|101| tool-eval final (full 69, TC-61 excl) | 86 | 85 | 87 | 86 | 87 | **90** |102| GSM8K (50q) | **98.0%** | 94.0% | **98.0%** | **98.0%** | **98.0%** | 96.0% |103| MMLU (50q) | 74.0% | 76.0% | **82.0%** | 80.0% | 76.0% | 64.0% |104| IFEval (20 prompts, prompt-level) | 90.0% | 90.0% | 90.0% | **95.0%** | 90.0% | 90.0% |105 106### Decode throughput — tok/s output (ISL 128 / ISL 512)107 108`vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4.109 110| Concurrency | Qwen3.6-27B (base) | Tess-4-27B | Tess-27B-RFI | **Tess-27B-RFA** | Tess-FP8 | Qwen3.6-35B-A3B |111|---|---|---|---|---|---|---|112| 1 | 57.0 / 61.4 | 59.9 / 63.8 | 75.3 / 69.3 | 67.9 / 58.2 | 86.0 / 85.1 | **91.9 / 114.4** |113| 10 | 304.5 / 233.6 | 289.6 / 227.8 | 281.5 / 219.2 | 292.2 / 194.8 | 280.6 / 269.2 | **434.9 / 440.3** |114| 25 | 424.4 / 321.7 | 492.0 / 341.1 | 429.1 / 284.0 | 369.0 / 260.3 | 533.2 / 355.4 | **688.9 / 563.7** |115| 50 | 556.0 / 349.8 | 527.9 / 345.6 | 452.8 / 295.0 | 422.6 / 278.4 | 560.3 / 394.1 | **889.8 / 702.9** |116 117### MTP draft acceptance by work category118 119Measured from live serving logs, k=5 draft tokens, drafted-token-weighted aggregation.120 121| Work category | Overall acceptance | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 |122|---|---|---|---|---|---|---|123| JSON generation | **87.3%** | 96.4% | 92.2% | 86.8% | 82.9% | 78.0% |124| Math | 78.2% | 95.6% | 87.7% | 78.6% | 68.8% | 60.2% |125| Code | 63.4% | 89.4% | 74.9% | 60.7% | 49.8% | 41.9% |126| Creative English | 63.1% | 88.0% | 73.3% | 61.6% | 50.1% | 42.3% |127 128## Notes129 130All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in.131TunableOp is untuned — GEMMs run on default heuristic-determined values.132Base Qwen3.6-27B figures are the 2026-07-12 re-measurement on the same tcclaviger/vllm:latest image and thinking-OFF methodology as every other build, replacing an earlier non-comparable run.133 134## Credits135 136- **Tess-4-27B** by Migel Tissera ([migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)) — the model quantized here:137 138  ```bibtex139  @misc{tissera2026tess4,140    title        = {Tess-4-27B},141    author       = {Migel Tissera},142    year         = {2026},143    howpublished = {\url{https://huggingface.co/migtissera/Tess-4-27B}}144  }145  ```146 147- codeneedle (positional recall) originally by Alexander Ziskind, expanded test suite by tcclaviger (Rob Smith).148- Tool-calling scenarios (incl. TC-61) run on tool-eval-bench by SeraphimSerapis (Tim Messerschmidt), scenario methodology adapted from ToolCall-15 by stevibe.149- GSM8K, MMLU, and IFEval run via tool-eval-bench's built-in accuracy benchmarks at their defaults: GSM8K 8-shot CoT, MMLU 5-shot, IFEval zero-shot.150