PPeSse/WhiteRabbitNeo-v2-q4f32-transcoded-MLC

WhiteRabbitNeo-v2 — q4f32_1 MLC (transcoded)

MLC / WebLLM build of WhiteRabbitNeo v2 (Llama-3-8B) in `q4f32_1` quantization, for running under WebGPU without the `shader-f16` feature.

Why this exists

The native q4f16_1 build (e.g. `pasmot/WhiteRabbitNeo-v2-MLC`) requires the WebGPU shader-f16 feature. On some setups that feature is not exposed — notably Chrome/Dawn on Linux with an NVIDIA Blackwell (RTX 5090) GPU, where Dawn brings up a Vulkan 1.1 device and shader-f16 (which needs Vulkan 1.2+) is unavailable. A q4f32_1 build uses float32 compute and does not need shader-f16, so it runs there.

How it was produced (important — this is a transcode, not a re-quantization)

mlc_llm convert_weight from the current MLC nightly (0.26) segfaults while compiling the quantization kernel, so this model was not re-quantized from the fp16 weights. Instead it was transcoded from the existing q4f16 build `pasmot/WhiteRabbitNeo-v2-MLC`:

• —q4f16_1 and q4f32_1 share the exact same 4-bit packed weights (*.q_weight, uint32) — these are copied byte-for-byte.
• —Only the float tensors differ: the group scales and the RMSNorm weights. In q4f32_1 they are logically float32 but stored on disk as bfloat16 ("format": "f32-to-bf16"), the same 2 bytes as q4f16's float16.
• —So each float16 tensor was decoded and re-encoded as bfloat16, and its dtype relabeled float32; byte offsets and shard sizes are unchanged.
• —mlc-chat-config.json was set to q4f32_1, context_window_size: 4096, prefill_chunk_size: 4096 → 1024.

Consequence: numerically this is equivalent to the q4f16 model (the scales carry q4f16's rounding, only promoted to f32/bf16); it is not a fresh q4f32 quantization from the fp16 weights. Quality ≈ q4f16, compute in f32.

Provenance

• —Base model (original fp16): `WhiteRabbitNeo/Llama-3-WhiteRabbitNeo-8B-v2.0` (Apache-2.0)
• —Transcoded from (direct source, q4f16): `pasmot/WhiteRabbitNeo-v2-MLC`
• —Quantization: q4f32_1 · Architecture: llama (Llama-3-8B)
• —Model lib (WebGPU wasm): Llama-3-8B-Instruct-q4f32_1-ctx4k_cs1k-webgpu.wasm from mlc-ai/binary-mlc-llm-libs (web-llm-models/v0_2_48)

Usage (WebLLM)

const appConfig = { model_list: [{
  model: "https://huggingface.co/PPeSse/WhiteRabbitNeo-v2-q4f32-transcoded-MLC",
  model_id: "WhiteRabbitNeo-v2-q4f32-transcoded-MLC",
  model_lib: "https://raw.githubusercontent.com/mlc-ai/binary-mlc-llm-libs/main/web-llm-models/v0_2_48/Llama-3-8B-Instruct-q4f32_1-ctx4k_cs1k-webgpu.wasm",
}]};