Modelpublic

ajh-code/Qwen3.8-27B-Escha-W2-16GB-120Kctx-KVarN

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

2likes81downloads

Model Card

<h1 align="center">Qwen3.8-27B Escha W2 · 16 GB · 120K Context · KVarN</h1>

Run the complete 27B Escha W2 model at up to 120K context on a single 16 GB NVIDIA GPU using a real packed KVarN K4/V4 cache.

No scored quality loss was observed in the current matched 45-point KV-cache suite; packed-attention numerical similarity is above 0.999999.

<a href="https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2">Original Escha model</a> · <a href="https://huggingface.co/EschaLabs/escha-runtime-qwen3dense">Escha SGLang runtime</a> · <a href="https://github.com/huawei-csl/KVarN">KVarN</a> · <a href="https://arxiv.org/abs/2606.03458">KVarN paper</a>

Download

Component	Purpose	Included here?
Packed KVarN backend	K4/V4 cache allocator, fused decode, graph integration	Yes
Setup and serving tools	Pinned venv install, launch, validation, benchmarks	Yes
Escha W2 model	Original 10.15 GB two-shard checkpoint	Downloaded by `setup.sh`
Escha SGLang runtime	Original compiled runtime wheel and serving fork	Downloaded by `setup.sh`

This repository intentionally does not duplicate Escha Labs' weights or compiled wheel. The root config.json is a Hugging Face model query file, and the recommended whole-repository download fetches it. This allows Hugging Face to track this package's downloads independently from the upstream model and runtime.

Versioning begins with v1.0 experimental. The implementation works; the hardware matrix and workload-quality matrix will continue to grow.

Quick start

Requirements: Linux x86-64, Python 3.12, CUDA 12.x, a working C compiler and Python development headers, and an NVIDIA sm_80–sm_120 GPU. The default 120K profile requires a 16 GB-class card and about 30 GB free disk space.

bash

hf download ajh-code/Qwen3.8-27B-Escha-W2-16GB-120Kctx-KVarN \
  --local-dir qwen38-kvarn
cd qwen38-kvarn

chmod +x setup.sh serve.sh preflight.sh validate.sh
./setup.sh
./validate.sh

CUDA_VISIBLE_DEVICES=0 ./serve.sh

Nothing is installed system-wide. setup.sh creates .venv locally, installs Torch 2.9/cu128 and the pinned Escha runtime, then downloads the pinned model.

The OpenAI-compatible endpoint defaults to http://127.0.0.1:30000/v1, with served model id escha-qwen38-27b-w2-kvarn.

bash

curl http://127.0.0.1:30000/v1/models

LAN serving must be enabled deliberately and should use authentication:

bash

HOST=0.0.0.0 API_KEY='replace-me' CUDA_VISIBLE_DEVICES=0 ./serve.sh

Do not expose an unauthenticated raw inference endpoint to the internet.

What KVarN changes

The Escha checkpoint already compresses the weights to about 10.15 GB. This project independently compresses the growing full-attention KV cache.

For this model's 16 full-attention layers, four KV heads, and 256-dimensional heads:

KV storage	Bytes/token across 16 layers	Relative to FP16
FP16 K/V	65,536	100%
FP8 K/V	32,768	50%
KVarN K4/V4 records	17,536	26.8%
KVarN plus shared FP16 layer workspace	21,632	33.0%

Completed 128-token history tiles use tight K4/V4 records. The first 128 tokens, unfinished live tile, and bounded generated tail remain FP16. Chunked prefill reuses one FP16 layer workspace rather than allocating a full FP16 cache.

Decode uses fused Triton split-K attention that reads packed K/V directly. Batch-one CUDA graphs keep the first 512 generated tokens in an exact FP16 tail. Outputs longer than 512 tokens automatically fold that tail into KVarN and continue safely in eager mode.

Measured performance

RTX 5080 16 GB

Matched single-stream runs use the same Escha W2 checkpoint, INT8-as-stored LM head, Triton attention, one request, no radix cache, and exactly 256 generated tokens:

Configuration	2K decode	16K decode	16K TTFT
Stock FP16 KV, CUDA graph	59.83 tok/s	50.55 tok/s	9.02 s
Stock FP16 KV, eager	44.75 tok/s	44.75 tok/s	8.98 s
Early packed KVarN eager path	35.08 tok/s	35.24 tok/s	9.52 s
Packed KVarN v1.0 + CUDA graph	57.42 tok/s	44.02 tok/s	9.62–9.69 s

CUDA graphs recover the fixed launch overhead: v1.0 retains 96% of stock graph decode speed at 2K and 87% at 16K. The remaining long-context gap is the real packed-attention work. Prefill remains about 7% behind the matched stock run because completed chunks must be transformed and packed.

RTX 5060 Ti 16 GB

The packaged release—not the development launcher—also completed model load, KVarN allocation, CUDA-graph capture, fused decode, and generation on an RTX 5060 Ti. A short 2K/16-token check measured 30.06 tok/s. This is a functional portability check, not a complete performance characterization.

Long-context and lifecycle testing

Test	Configuration	Result
Near-limit capacity	Exact 119,900-token prompt	Completed in 136.7 s; pool reached 97%
Near-limit retrieval	Exact 118,038-token prompt; secret at beginning	Retrieved `banana` in 134.8 s
Repeated long requests	Two consecutive exact 64,000-token prompts	Both retrieved `banana`; 54.40 s and 54.82 s
Graph request reuse	New request after graph decode and allocator wrap	Correct reset; no stale history or CUDA fault
Graph-to-eager boundary	2K prompt, forced 600-token output	Completed all 600 at 53.66 tok/s average
Short reuse	Consecutive 2K requests	Correct reset; approximately 58 tok/s on RTX 5080
Chunked prefill	4,096-token chunks through 64K	Stable; no false mid-request reset
Oversized chunk control	8,192-token prefill chunks at 120K profile	Rejected as unsafe after measured activation OOM

The release therefore defaults to 4,096-token chunks, a 120,000 total sequence limit, and a 120,512-slot physical pool. The sequence limit includes generated tokens; applications must reserve output space inside 120K.

KV-cache quality comparison

The matched quality suite used the same W2 checkpoint, one RTX 5080, INT8 output head, Triton attention, one request, radix disabled, temperature 0, and seed 20260824. KVarN used the real g128 K4/V4 codec mathematics. The proxy was used for the broad paired suite so every cache arm could be compared under the same stock attention implementation; the separately tested packed fused path is covered by the numerical gates below.

Case	Prompt tokens	FP16	FP8 E4M3	KVarN K4/V4
Short closed questions	390	10/16	10/16	10/16
Distributed retrieval	11,957	12/12	12/12	12/12
Distributed ledger	11,912	1/4	1/4	1/4
Dependency rules	7,972	3/4	3/4	3/4
Python comprehension	8,080	0/3	0/3	0/3
Filtering and sorting	4,076	0/6	0/6	0/6
Total	—	26/45	26/45	26/45

No scored FP16 pass became a KVarN failure. All three formats retrieved twelve values distributed across a 12K prompt exactly. Each arm was repeated and reproduced all seven outputs and captured token log probabilities exactly, so the cross-format comparisons were not sampling noise in this setup.

The equal 26/45 result means no measured score loss in this suite, not universal proof that KVarN is quality-neutral. The model itself missed the same arithmetic, code-tracing, and sorting items under FP16. These thinking-off results must also not be compared directly to Escha's thinking-on public benchmarks.

Where text trajectories differed, KVarN was closer to FP16 than FP8 in the small comparison:

Case	FP8/FP16 output similarity	KVarN/FP16 output similarity
Ledger	0.9878	0.9878
Python comprehension	0.3538	0.8824
Open technical generation	0.5872	0.7484

Packed-kernel numerical validation

./validate.sh runs the codec, packed decode, exact graph-tail merge, and causal prefill comparison on the selected GPU:

Gate	Reference comparison	Result
Codec K round trip	Random FP16 K tiles	cosine 0.9950774
Codec V round trip	Random FP16 V tiles	cosine 0.9942523
Fused packed decode	Explicit dequantization + FP32 attention	cosine 0.99999988, max abs 0.000292
Exact graph-tail merge	Explicit dequantization + FP32 attention	cosine 0.99999982, mean abs 0.000047
Causal packed extend	Explicit dequantization + FP32 attention	cosine 0.99999970, max abs 0.000475

These tests cover packed nibble extraction, K/V metadata, Hadamard rotation, KVarN normalization, grouped-query head mapping, split-K online softmax, log-sum-exp partition merging, dynamic graph-tail length, and causal masking.

Validated scope

Area	v1.0 status
Escha W2 weight loading	Complete two-shard checkpoint through pinned Escha v1.2 runtime
Persistent KV allocation	Real tight K4/V4 history for all 16 full-attention layers
Decode	Fused packed Triton attention enabled by default
Prefill	Exact current chunk through stock Triton; pack after use
CUDA graphs	Batch one, exact 512-token generated tail, bounded eager fallback
Context capacity	120K configured; 119.9K capacity and 118K needle probes completed
Request lifecycle	Repeated 2K and 64K requests, allocator wrap, graph reuse
Broad quality gate	45 scored checks across FP16, FP8, and KVarN codec arms
Hardware	Full stress/performance on RTX 5080; packaged launch/generation on RTX 5060 Ti
Security	Localhost default; API key supported for deliberate LAN binding

Compatibility and limits

—Intended GPU range: NVIDIA Ampere, Ada, Hopper, and Blackwell (sm_80 through sm_120), matching the compiled Escha runtime.
—Not supported: pre-Ampere cards, non-NVIDIA GPUs, Python other than 3.12, CUDA 11, or native Windows serving.
—Only one active request is supported. Batching and prefix/radix caching are refused rather than silently producing incorrect state.
—The cache layout is specific to this model: 16 full-attention layers, four KV heads, head dimension 256, and 128-token groups.
—Only RTX 5080 has completed the full near-limit and reuse stress suite. RTX 5060 Ti has completed packaged launch and short-generation validation.
—Direct packed prefill is included for research but remains off by default: it measured much slower than materialize-then-Triton on the RTX 5080.
—This is a monkey patch against a pinned Escha SGLang fork. New upstream runtime versions must be revalidated before changing the pinned revision.

Useful overrides:

bash

# Another physical GPU
CUDA_VISIBLE_DEVICES=2 ./serve.sh

# Smaller context profile
CTXLEN=65536 MAXTOKENS=66048 ./serve.sh

# Eager debug path
GRAPHS=0 ./serve.sh

# Ampere single-stream users should A/B the alternate Escha route
ESCHA_ROUTE=blackwell ./serve.sh

Reproducibility

setup.sh pins:

—Model revision: a7329bde673acf42826f17bf525154e633006cbd
—Runtime revision: 36bb7bf5faebd4dbe636628ccd087ad5ecf6d0ac
—Python 3.12, Torch 2.9.x, CUDA 12.8 wheel index

Override MODEL_REVISION or RUNTIME_REVISION only when intentionally testing a newer upstream release.

Validate the download

Before installing dependencies, verify every distributed file:

bash

python3 validate_release.py

MANIFEST.json records the size and SHA-256 of every release file except itself. After setup, run the GPU and numerical suite:

bash

./validate.sh

License and attribution

This integration is released under Apache-2.0. The KVarN codec equations, record layout, and fused-attention structure derive from Huawei CSL's Apache-2.0 KVarN implementation. The original Escha model and compiled runtime are downloaded from Escha Labs and retain their upstream licenses and third-party notices. See `NOTICE.md`.