abhishek085/spark-s1-4b-v6-nvfp4

spark-s1-4b-v6-nvfp4

NVFP4 (MLP-only) post-training quantization of `abhishek085/spark-s1-4b-v6`, part of Open Spark Jev. Only the MLP gate_proj/up_proj/down_proj weights are quantized to NVFP4; every attention-related module (both the 8 full-attention layers and the 24 linear-attention layers — their conv1d, gating and projection weights) plus lm_head stay in bf16, since the option-letter logit readout needs full precision there.

This is a very early release; expect it to change quickly.

Speed vs accuracy

Measured on one NVIDIA DGX Spark (GB10), batch size 1, served via vLLM.

1.40x faster with no measurable accuracy cost — the small standard-tier dip (1.000 -> 0.972) is more than offset by a hard-tier gain (0.595 -> 0.622, within the noise floor for 111 items). This is a much cleaner quantization result than spark-s1-4b-v5-nvfp4, whose same recipe cost about 1.9 Intelligence points on the uniform Qwen3-4B backbone.

JevBench v1.2, public items only (231 of 534). This is the only accuracy check run on this variant so far — not own-splits/external tested independently (inherits spark-s1-4b-v6's own numbers there as a reasonable prior, given the near-zero JevBench cost, but hasn't been directly measured).

The smaller speedup than v5-4b-nvfp4's 1.66x is expected: only the MLP is quantized, and this hybrid backbone carries relatively more of its total compute in the unquantized attention/linear-attention paths. Even so, spark-s1-4b-v6-nvfp4 (53.3ms) is faster than spark-s1-4b-v5-nvfp4's own base model spark-s1-4b-v5 bf16 (61.6ms) while being far more accurate.

How to serve

docker run --rm --gpus all -p 8355:8355 vllm/vllm-openai:nightly-aarch64 \
  --model abhishek085/spark-s1-4b-v6-nvfp4 --served-model-name spark-s1-4b-v6-nvfp4 \
  --host 0.0.0.0 --port 8355 --max-model-len 8192 --trust-remote-code

Then point the repo's gateway at it in OpenAI-backend mode:

python -m open_spark_jev.serve.gateway --backend openai --upstream http://127.0.0.1:8355/v1 --model spark-s1-4b-v6-nvfp4 --port 8472

--max-model-len should be at least ~4-8k tokens: some Jev-style decision states (long documents, tool traces) run well past 2k tokens.

Reproduce this quantization

pip install nvidia-modelopt   # a current release (0.46+); the TensorRT-LLM container's bundled version predates Qwen3.5 support
python scripts/quant/ptq_nvfp4.py --model checkpoints/v6-4b --tokenizer models/Qwen3.5-4B \
  --calib runs/quant/calib_prompts_v5.jsonl --out engines/spark-s1-4b-v6-MLP_ONLY_CFG --cfg NVFP4_MLP_ONLY_CFG

Calibrated on the same 980 real decision prompts used for spark-s1-4b-v5-nvfp4 (v6 trained on identical data, so these are directly reusable).

Limitations

Inherits spark-s1-4b-v6's limitations (Jev-style scope only, no RLCD, option-order sensitivity, English only). Additionally:

• —Calibration temperatures in calibration.json are inherited from the bf16 checkpoint, not refitted on quantized outputs.
• —Only JevBench-checked so far — own-splits and the 5-set external battery have not been independently re-measured on the quantized weights.
• —Needs a GB10/B200-class (Blackwell) GPU for the NVFP4 tensor core path.

License

Apache-2.0. Base model Qwen/Qwen3.5-4B (Apache-2.0).