Modelpublic

KIEFERSA/Sophea-Titan-1-NVFP4

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes20downloads

Model Card

<div style="display:flex;align-items:center;gap:18px;padding:16px 20px;margin-bottom:16px;border-radius:14px;background:linear-gradient(100deg,#e8f5ec 0%,#f7fbf8 62%);border:1px solid #c6e7d1"><img alt="KIEFERSA" style="flex:0 0 auto;height:58px;width:58px;border-radius:12px;object-fit:contain" src="data:image/webp;base64,UklGRgQNAABXRUJQVlA4WAoAAAAQAAAAxwAAxwAAQUxQSIsKAAABsMf/nyE52q+qZjd2nluEh00OsY2zbdsOH51t27Zt2744F1vrrqrf69X1m+mumq4K/ooISG4bSZKUmIIOeag1w1Ud8wFwPDgH2Om9zYFD2AcXDIa/hdWdgAff+r6oNK7pDDzw1uPxRlQRVoccFlu3e+pj0zrksByD8huqjSFSAm0c2l+8ilrIEQxaT1qC2ljAYYJD89PnxaYQQ05spcdMp5ZEcI0f/CtqGVsygTXY4+vYJGLI4QJg7CeIxkIOFwyGvE0tLeG0Xs8rVLGFHBZb1aORCXAYcniOQdfb61FJjXaEcEb5NeuoWRI+63DhShrgrAmdtZy4iAa4kJPj0PSMOdRCjuCQO3oaDXAhR3Dg+/9Ko0/IERxgl29pDA05QgCM+ohGn7DDYPDr1EJPr2cUiT7hp5aEhvUBZMZ6AY3rC3BjkLVdga0PkFjTGUqCDykOMw9kwEVwKIL9c1zX1d/saIJUsCFvE4sntILyFYjvjgQmeFBwbCsuaAMCyldhhOqVAYERp7bm2s3MKTMoKTF6apugiEOrvaMr+XbADJq8JtXf391IQImt4aEqYpBH3kmtuaXS7E4gkRL1s73pBS8IEY1rLusYCHFi+PoQurwnkjdh+dQ2wHP+i71p/GgM/XaQggITFpzdIpaQoWL7ZlcaGtKD5LPnrBOaeB8tbO3XAzg5LylIsYe/HyY8FwuLNE4/uoSck5SkkO/38jsk2djc05uSK5uSlOvQZzsAcBESdGyLJrckP4hZkFLeGw7AuZ+kDnAXt6fVyIa0ol7u62u0SGfrri2nM5xDQpJ8oqefksbq7+hOk4Eddgmw4e7uwHLMa6TC6NEe9DLaYyfV13sY95JMv5AX4FxgK6sv7ZiFCVwkDFdIifj2EBp93GAvS6a29ivuFbRPx1Jzi135nndWsyKXMdZvxLDCo7kDlNT43Z5AjjYTFDiW6ceVFFEYNJmOqtCthdgLuCXSBLiDaIBzi5sJv5PjKRalf6PU+UNptZ0lKrZpx5WSo8wA6eLeHhBLkZiWsNGWG/P8nEmfnwySd499Og5AsMyDSye3ISuzexyKQnx7KGSfSzvS6FMEHIt+ui2wjEPrffFxUcaGg8g09IHJPqg1DgGe9Q34AeLQTSF0Wr5egOfMJlKBh5S4QQ+f9zNqqfwkS9bzMV3TCU4yHXsq1LDYut9Ti7i6G0Dzc0gHpX9kxcpvXIeqEas7QQmDNv9bhipSgYUE2PaXrUIVadJHwUhv7pWrUUU6oJAvHi0nLzUnK78fhJ7STrdWo4rCSWzNzphvFpSkvhYuGGxxv+ldDyM5DqXHzEAtdZr+HBZLr6ckShk+BAdx0O+mTCX1GSUUslcwFk8oprE9v0EtpU2/lBAAI99DLWW44AJg/EfmzKTr+0qSnb4y0SJMCMFg2FsapXTRvyY48L1/MhPCA4+t9wsKpXTVhyc4iCOmmZAUFlhsVY81oops+glTSZNTZpv1NBzw2LreVYdKard9kSyWFhMWmLgXAmg+KLtuHSpZhP5OJhi0vWC5CUn+w3Ic2l9IAhzaYXfyr1pjdttzBIfWE5bQ/OwAiwldbiMPQyZxGeBO+we1MQdYylYPNpoJGcSZ5Y6ZRs0BDhao7Z5VqKSPCA7soLz3fUc4kIFvIErpG5wD7JYX4LIB/WFw9IdGMkRfKM1ZjPxgNuZj+tOCS9xck12/NtckM6T+HSwh8r9F7z23uHpGDjTPiMwCiPjQ1TdcbzGM9XqOrgWOcSe5o/5CFRURdyNvbZYa3eO0hsxGHJIFZGQ1utxeRz+2uMettJywbCSIjG9WUssIViGpvD2wjKONFQfXAgCQdRB9AICxzIO+ALAhy7ouwNYHSB1vSsKKRSvuzsAEDx8OOgWnbr38jQFG/CC7tvaazaB8BTY+uV0RGkkg43nIsdXe0RkEVKzECOvv28JMCMbGsTU8vCUwIaB8FfkAUn1LJbAc84mbJ/33PxbD2eIoUT5rbmGe10dBH9ErOwITzBM02vYTOjKF+PpQc/sW7AchE5b9p7VLyfj7tKMZ+CFpxU3ua9GRxoVnN3fXHpzx7xsuTOPXuwFpnEyETNA4+/gmwEXoULH9sj8n5yeBhD3745CSWHzHQQvzkSXAcwCJJO7dD3uzWEKFMq24pzUl5yUVifLpjgBChAgd25KJLcnVTUWqNejdkaYgBwZSlVZc0o7U15SkrGwv9TfiK+ltzfVltCKlJW11i57sacRPUge4O7tScwpipLD+/m5mh4OAVBg90oNeTlvSnvrqGyqM+I6WCvWzfelN6Yo0O77mso4muHqNlIhvDqXRxwU2E5ZNbm1CkrdIqfGTsdTcYFcq5p3ZHLjwExXbd7vRIuoK29I9/YQmsfiHijT+diAnR+wKF/LrISIWbyh03Y7LkaMtOkn31V4mJHlC3oow70wa4IpOitXq0/EAnHsBscVTWpF1zS3u5O2hRjwg0rjq4va0YjrGoegX+5jDyv6m+vpKenjOcSgK5WNVkH3u7kZrvnPcf6q6vSWwjEPLbVFwLsNAZAJbyz6oNQ4BnvmNHyAO3RRCp+XrBXgOylZiowo8pFBvddMlc1BL7SdZsorra+sqm/5vuZFQw3IcOly2EnFNV4COl5Om2gBDvni0nLIEVSNWd4ISBpW31hjJNn+j1PlDaRWTDWt21jyTbEgfBeOCQff761FGOqib2EqOm2XyWaF+ECPbPtWISnpF8Y0f+DtqqQr1tRRoqn1Vo5TucC7de/YoPEozYHt8h1qqpP4cKgCj3sVYHOH14AJg3MeopUzbZyQEwM5foJYqk/CkUWwb/qa5zjb9UoID3+dH1FKGa8MFg74vmOfOtu9LcMgd9reZECZYbD0fbUQVpe1fS5QmJ80166gd/lrXu2tNhU3Zh5cqYzQ/b6GRoEC+eJRdvxaV1O76CZlg0OaC5SZaeEyqi9ruEtJQbNUXmVI6Xk3iXiAQDFpNWYramAUWt08nGvcCQDyj2anz6NuANaknVD3QgCryHcGh9OgZtJxaYlkkej2tUEmfEQLEAb/QUOAASxn0GqKUviI4wO7f0EBQLAqHpFEfGfERYQJc3jEXHwATwnb+2lwL3+CCweA3dGwai0tS3Nv/F1OQC+CJ9Xpe0hcYd7gRceTfRnyBxbblIzTAOcRd7Why2hwzgeDBjM531JHo4xSXnxNaTFiEKvIABmXXVNNPR25xm63aX7AccUjmaX/BSmqucZ51r6kbASLjzKMBrhi4lqoyYBkHpUbMOgCMzMg4CtEHADjLPOgLABuyrOu0fkDpmi7rAUi2WfgvYIGHvNnV3do5SH/gfEbjw1saCxGOZ6hntzWZM+xIifqNwdTChEPDD8cA8NgCDnnL/nJXaiFHxfbL/oy0f4YMNzbtqFJiIUdFCv85tTmxgEPeTxdPaEUs5OhI4coL2pJ/WAghdrbu2jLyNSqMw8Zq7+pCLeSYAPdYFbWQIyXK5/rQ6BNypER8fQi1oJLKPhoPYCzkKKnxm92AFKrgkmi/HSiIBZgEm3F0KbGQoyKN805tSizk6EjhkqktSfQJNvkB7pJ2wIyFG2JrbyynoSHkmC9hd3WjFnYweqQnDQ2h59W+1IIPAABWUDggUgIAAHAdAJ0BKsgAyAA+MRiJRCIhiBJcEAGCWlu4MAAZsUGEUQU6Vr/HVACel73mNKbZii7/0o6X9Y+s6FoAm3srqdXIcWd/45kb7mdOviNuuq60W3qXx/nI9sR42ICkrOwkSyYty0cun8cMpqXF0fh8fW3dSwaV0akfevRFNfnTLgy9bQM6IEWlj9O95hHRhKjNbDCS4qWJnQbJto47spFiqntPEoRO945gp7f+FzLmeAJQakEjD+xgQDxMOYrHw79WEkLQkKplqTo/uTX4n54eEP0N4epeSNwNE7Cl7q0Xi32YrSRma3RieLfasZELkHnrVj60WkAA/uouoblbKpSNm/wu8g0tISmLMdPrSIKVi8oMaNvyXkZc6zskreB0cAODf99ubzUwj0N8IJywGgCWpByLV1Kx+G8TMEVJa0HKGlyheflfEABBIEOGipfrRMJEDIiV7tNyTCRkHRlacBBPJNrPF6gu5KqHk0mGh2xBZYDKEWXI1A8R9WyRig0s6CIELwgJcqvHXXixg1yTYYd3Ja2rm0FNGHOb8MdNErR6nH3DYmradAb7JnP/RNqQ3gvhl/pA7ygUm6CIWAO/JF0d93ZViT6KqEgvSVdzZIwTzPR1TQ5YDMPK1u42qqN2SqTRwtzHrtold7EeIrx7plafSjlNtoSSwuC99MOigEHtk+7uPfbMdQA6uO4+wAsQ6bfVGy+dpyM63pbfBdYa2IbgqA8ZA3EUsyDx6GwlVker0XVZhE/HSCyOiMAVqYYDYXpIECKSg6k4BD3OKMJ7nJlAAAAAAA=="/><div><div style="font-size:1.24rem;font-weight:700;color:#003713;line-height:1.2">Sophea-Titan-1-NVFP4</div><div style="font-size:.86rem;color:#3a5a47;margin-top:2px">NVFP4 4-bit quantization of Sophea-Titan-1</div></div></div>

NVFP4A16 (weight-only 4-bit) quantization of KIEFERSA/Sophea-Titan-1 — the 27B Greek fine-tuned chat model built on Qwen3.6-27B. This is the smallest Sophea-Titan-1 variant — ≈0.35× the disk / VRAM of bf16 — at a modest, measured accuracy cost (see below). For a lossless drop-in, use the FP8 build KIEFERSA/Sophea-Titan-1-FP8 instead.

—Base model: KIEFERSA/Sophea-Titan-1 (bf16)
—Scheme: NVFP4A16 — weight-only 4-bit NVFP4 (E2M1) weights + FP8 block-scales, 16-bit activations, compressed-tensors nvfp4-pack-quantized
—Size: 18.8 GB (bf16 base ≈ 54 GB → 0.35×)
—Creator: Kiefer SA
—Decoding: non-thinking (enable_thinking=false) — see recommended sampling under Usage

What is / isn't quantized. Data-free one-shot PTQ via llm-compressor (NVFP4A16, no calibration). Quantized to 4-bit: the language-model attention + MLP Linear layers. Kept bf16: lm_head and the two GDN gate projections linear_attn.in_proj_a / in_proj_b (out-dim 48 → fused 96; kept full-precision so vLLM's Marlin FP4 kernel — which requires output dims ÷64 — can serve the model). This quantized build is text-only — the vision tower of the multimodal base (Sophea-Titan-1) is not included in this NVFP4 release.

Serving. NVFP4 runs through vLLM (Marlin FP4 kernel on Blackwell/Hopper). Unlike FP8, the nvfp4-pack-quantized format does not load in HF transformers — use vLLM. Serve non-thinking (enable_thinking=false).

Accuracy vs bf16 (measured on this build)

Measured through the vLLM decode path (greedy, non-thinking), vs the bf16 base:

The 4-bit weight quantization costs ≈2.3 pt on General Greek benchmarks (concentrated in Greek-knowledge benchmarks), while English retention is nearly intact (−0.9 pt) and generation stays degeneration-free. This is the size/quality trade for a 4-bit footprint; the FP8 build is lossless if you can afford ~30 GB.

Per-benchmark detail (measured on this NVFP4 build)

General Greek benchmarks (9, accuracy)

English retention — 5 benchmarks (accuracy)

greekmmlu — per-subject (accuracy, 30 subjects)

<div style="overflow-x:auto"> <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> <thead><tr> <th style="padding:10px 8px;text-align:left;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;">subject</th> <th style="padding:10px 8px;text-align:center;font-weight:700;border-bottom:2px solid #0a8043;color:#00682f;font-size:14px;background:rgba(10,128,67,.10);">NVFP4</th> </tr></thead><tbody> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Accounting</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.848</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Agriculture</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.846</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Art</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.760</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Biology</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.847</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Chemistry</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.741</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Civil Engineering</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.791</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Clinical Knowledge</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.807</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Computer Networks & Security</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.651</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Computer Science</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.866</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Driving Rules</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.803</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Economics</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.897</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Education</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.871</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Electrical Engineering</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.801</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">General Knowledge</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.769</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Geography</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.936</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Government and Politics</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.935</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Greek History</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.841</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Greek Literature</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.500</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Greek Mythology</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.840</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Greek Traditions</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.846</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Law</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.665</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Management</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.812</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Maritime Safety and Rescue Operations</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.662</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Mathematics</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.901</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Medicine</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.841</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Modern Greek Language</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.901</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Physics</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.835</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Prehistory</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.968</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">World History</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.950</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">World Religions</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);background:rgba(10,128,67,.10);">0.761</td> </tr> <tr> <td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;">OVERALL</td> <td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#00682f;background:rgba(10,128,67,.10);">0.832</td> </tr> </tbody></table></div>

Usage

Serve with vLLM (compressed-tensors NVFP4 → Marlin FP4 kernel):

bash

vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Serve with vision enabled — this is a multimodal checkpoint (Qwen3_5ForConditionalGeneration). The vision tower is excluded from quantization and kept in bf16, so image input works unchanged:

bash

vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \
  --limit-mm-per-prompt '{"image": 4}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

For text-only serving, add --language-model-only to skip loading the vision tower entirely.

python

resp = client.chat.completions.create(
    model="sophea-titan-1-nvfp4",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        {"type": "text", "text": "Περίγραψε την εικόνα στα ελληνικά."},
    ]}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)

Recommended sampling (instruct / non-thinking): temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-titan-1-nvfp4",
    messages=[{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)

Speculative decoding (MTP)

This build ships the model's multi-token-prediction head — 15 mtp.* tensors (~0.85 GB, bf16) in model-mtp.safetensors. This is the single-layer draft stack that config.json has always declared through mtp_num_hidden_layers: 1.

Earlier revisions of this repo did not contain it. The LoRA merge loaded the base through AutoModelForCausalLM, and transformers declares _keys_to_ignore_on_load_unexpected = [r"^mtp.*"] for this architecture, so the head was discarded at load and never written back — and llm-compressor dropped it again during quantization. The config therefore advertised a module the weights did not contain. To pin the previous bytes, use revision="f3e59d294e45e68870fc157c06d04cdb6c322691".

The MTP Linears are held in bf16 and listed in quantization_config.ignore, so the compressed-tensors loader treats that block as unquantized rather than expecting NVFP4-packed weights. (vLLM's built-in bf16 exception for mtp.fc applies only to modelopt_fp4 checkpoints, not to compressed-tensors, so the explicit ignore entries are what make this load.)

Provenance. These are the base Qwen3.6-27B MTP weights. LoRA never targeted mtp.*, so no fine-tuned draft head exists. This cannot affect output quality: speculative decoding verifies every drafted token against the main model, so a stale drafter changes throughput only, never the output distribution.

Enable it with vLLM (≥ 0.23.0):

bash

vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

The head is loaded but stays inactive unless --speculative-config is passed, so existing serve commands are unaffected.

License

Inherits the Qwen3.6-27B base-model license. Verify base-model terms before use.