P4pps3n/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw

Laguna S 2.1 EXL3 2.65 bpw HQ with 8 bpw DFlash

This repository contains a low-VRAM EXL3 deployment bundle for poolside/Laguna-S-2.1, paired with a separately quantized EXL3 version of poolside/Laguna-S-2.1-DFlash.

The tested deployment target is one concurrent sequence with a 160,000-token Q8 cache on exactly two RTX 3090 GPUs. It uses ExLlamaV3 native tensor parallelism, DFlash model-draft speculative decoding, and no CPU MoE offload.

Quantization

The target contains 11 safetensors shards. The DFlash checkpoint is stored in draft/ and uses draft_mode: model; it is not an integrated MTP checkpoint.

For Hugging Face schema compatibility, root config.json reports integer quantization_config.bits: 3. The exact nominal target, allocation, and packed tensor metadata remain in quantization_config.json. The model is not a 3-bpw conversion; its measured whole-safetensors rate is 2.724383 bpw.

Required ExLlamaV3 tensor-parallel fix

Laguna requires softplus per-head attention gating. ExLlamaV3 1.4.1 omits gate_softplus from SlidingAttention.tp_export(), causing TP-imported modules to silently disable the gate and generate repetitive garbage even though model loading succeeds.

Apply runtime-patches/exllamav3-laguna-tp-gate-softplus.patch to the exact ExLlamaV3 1.4.1 checkout before serving with native tensor parallelism. The patch preserves self.gate_softplus in the exported constructor arguments.

The fix was isolated using a known coherent Laguna 3.00-bpw checkpoint:

• —Native TP without the patch: repetitive garbage.
• —Legacy autosplit on the same two GPUs: coherent Python.
• —Native TP with the one-line patch: coherent Python.
• —This 2.65-bpw target with patched native TP: coherent Python and tool calls.

Do not treat a successful load as sufficient verification. Run a deterministic generation test before deployment.

Tested two-RTX-3090 profile

The provided tabby_config.example.yml uses:

• —One active sequence
• —160,000-token target cache
• —Q8 target and draft caches
• —Native ExLlamaV3 tensor parallelism
• —No CPU MoE offload
• —8-bpw EXL3 DFlash in model-draft mode
• —RTX 5070 Ti excluded by CUDA_VISIBLE_DEVICES

Set model_dir and draft_model_dir to match your download location. The draft is inside this repository under draft/.

Measured runtime results

Hardware: two RTX 3090 GPUs connected through PCIe PHB topology, without an active NVLink path.

HumanEval used the official OpenAI 164-problem evaluator, one deterministic sample per problem, temperature 0, thinking disabled, and a 1024-token output limit. Generated code was evaluated with 8 workers and a 10-second test timeout.

Context sweep

The following values come from TabbyAPI server metrics after a fresh server restart. Each row is one cold-prefix run with temperature 0 and thinking disabled. Decode speed includes the separately quantized DFlash draft.

DFlash acceptance is prompt dependent. A separate short coherent coding test reached 46.43% acceptance and 106.89 tok/s; the repetitive long-context corpus above produced substantially lower acceptance.

VRAM warning

This exact 160K Q8 target-plus-DFlash profile passed the full operational prompt and all three cold starts, but GPU 0 reached only 18 MiB free at peak. This is a very small safety margin. Other drivers, display attachment, CUDA allocator behavior, or background GPU consumers can cause OOM. Stop unrelated workloads before loading, and reduce context if your placement differs.

TabbyAPI allocates cache in pages and reserves generation/speculation capacity. A request with nearly 160,000 prompt tokens can therefore be rejected even when the configured cache pool is 160,000 tokens. The validated operational prompt contained 154,057 encoded tokens and allocated the full configured pool.

Installation

Use the ExLlamaV3 1.4.1 CUDA wheel matching your Python, CUDA, PyTorch, and ABI, plus a compatible TabbyAPI checkout. Apply the included patch to the ExLlamaV3 source checkout and ensure that checkout is first on PYTHONPATH.

git clone --branch v1.4.1 https://github.com/turboderp-org/exllamav3.git
cd exllamav3
git apply /models/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw/runtime-patches/exllamav3-laguna-tp-gate-softplus.patch
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"

Restrict CUDA visibility to the intended two RTX 3090 devices before starting TabbyAPI. Prefer GPU UUIDs rather than numeric indexes.

export CUDA_DEVICE_ORDER=FASTEST_FIRST
export CUDA_VISIBLE_DEVICES="GPU-UUID-3090-A,GPU-UUID-3090-B"
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
python /path/to/tabbyAPI/main.py --config /models/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw/tabby_config.example.yml

Provenance

• —Target source: poolside/Laguna-S-2.1
• —Target revision: 00af5a51782109b587a3b3bbf11875e566036fa7
• —Draft source: poolside/Laguna-S-2.1-DFlash
• —Draft revision: b0486d1586daa0d56435c508108171fc1c8daff9
• —Quantizer/runtime baseline: ExLlamaV3 1.4.1
• —Target calibration: 250 rows x 2048 columns
• —Target output head: 6 bits
• —Draft nominal rate: 8.0 bpw

License

The model materials retain the upstream OpenMDW-1.1 license. See LICENSE.md. The one-line runtime patch is provided only to preserve the upstream model architecture flag during native tensor-parallel import.