Modelpublic

WaveCut/YuE2-3B-OrbitQuant-W4A4

sourceHugging Facecc-by-nc-4.0updated 9d agoView on Hugging Face

5likes3.2kdownloads

Model Card

YuE2-3B · OrbitQuant W4A4

Music generation from style and lyrics (m-a-p/YuE2-3B) with packed 4-bit transformer weights and activations. Output: 48 kHz stereo.

Model

—392 transformer projections in OrbitQuant W4A4 (224 packed modules). Embeddings, norms, auxiliary projections and the audio VAE stay in source precision; output heads are INT8 or W4A4 only in the profiles below.
—Checkpoint 3.04 GB vs 7.26 GB BF16 (−58.2%). Separate 0.53 GB VAE unchanged. No fine-tuning or distillation.

Run

bash

hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --mode fast --output out

One request per process is the default. A long-lived process serving many requests adds --warm-compile (compiled VAE decoder: 0.2 s instead of 0.7 s per request after a one-time 2 s cache load).

`--mode`	Changes	Logits
`off`	reference decode path	reference
`exact`	lookahead decode loop, ABC head cropped to the sampler's classes, model resident during VAE decode, compiled VAE decoder	bit-identical to `off`
`fast`	+ KV handoff to NAR, compiled NAR pointwise ops, INT8 output heads (DP4A GEMV)	KL 0.0015 semantic / 0.0002 ABC
`lowmem`	+ INT8 token embeddings	as `fast`
`turbo`	`lowmem` with W4A4 output heads	KL 0.016 / 0.0009

Requirements: Linux x86_64, Python 3.12, PyTorch 2.10.0 + CUDA 12.8; the OrbitQuant 0.9.8 wheel is bundled in src/wheels, native kernels 1.0.5 under src/runtime/kernels. Kernels: SM89 (validated on RTX 4090) and SM120 (built, not run). The first request compiles the VAE decoder (~25 s, cached afterwards).

Measurements

RTX 4090 (48-core host), identical input and seed. "Warm" is the second request of a long-lived process with --warm-compile; "process" is one src/run.py call from interpreter start to the written files (imports and model load included). Peaks are for the whole process.

Mode	Audio	Warm request	audio s / wall s	Whole process	NVML peak	torch peak
`off`	168.9 s	17.94 s	9.42	—	5.39 GiB	4.31 GiB
`exact`	168.9 s	15.85 s	10.66	21.6 s	5.33 GiB	4.05 GiB
`fast`	140.7 s	12.26 s	11.47	18.0 s	4.64 GiB	3.37 GiB
`lowmem`	152.2 s	13.49 s	11.28	19.3 s	4.47 GiB	3.37 GiB
`turbo`	178.2 s	15.57 s	11.44	21.7 s	4.71 GiB	3.56 GiB

exact keeps the off tokens and logits bit for bit; its waveform differs from off only at FP16 rounding in the VAE. INT8/W4A4 heads change sampling at the same seed, so compare those modes by audio seconds per wall second. Of a whole process, about 4 s are Python and library imports plus model load; the rest is generation. Wall time also depends on the host CPU: on a slow 4-core host the same exact request took 17.7 s and off 28.3 s. Raw data: evaluation/profiles/.

Files

src/: runtime, kernels and kernel sources; src/kernel-binaries.json lists the binaries per architecture, src/runtime-lock.json pins versions, SHA256SUMS covers every file. examples/: prompt file and WAV demos (BF16, W4A4, fast INT8 KV).

License: CC BY-NC 4.0 for the weights (inherited from M-A-P / YuE2); code components keep their own licenses. No commercial rights.