abhishek085/spark-s1-4b-v5

spark-s1-4b-v5

spark-s1 is an open, local "System One" decision model from the Nokast AI community (open-spark-Jev). It answers typed questions about a state (choice, boolean, ordered score, over options defined at request time) with a full probability distribution and a confidence from one forward pass, with no generated text. This is the accuracy pick of the v5 release: Qwen/Qwen3-4B fine-tuned with LoRA r16 (merged into these weights).

It is not Jev and is not affiliated with TypeSafe AI: it is a public backbone fine-tuned with a supervised loss. It follows the same contract (typed questions in; per-option probabilities, confidence out; no decoding).

What's new in v5

v5 narrows scope on purpose: training and evaluation are now Jev-style decisions only — agent-harness control (tool-call verification, context pruning, response-quality checks, delegation, error recovery, plan alignment), guardrails (prompt-injection and message-manipulation gating, PII/output-leak checks), moderation and ticket routing, retrieval/re-ranking, and structured records decisions (entity resolution, fraud and risk scoring, invoice/insurance/SOC routing). General-purpose text classification is no longer a training goal for this release. Training data grew from v3's 967 rows to 11,792 rows / 19,568 examples across 49 task packs, drawn from os-datagen (LLM-written scenarios, independently verified) plus new code-only rule packs where both the label and the surface text come from deterministic rules, no LLM in the loop. Recipe: LoRA r16 (merged), lr 5e-5, 1 epoch (lower learning rate and fewer epochs than v3), cross-entropy plus a Brier regulariser.

How it reads an answer

The state is rendered once; each question is appended as a menu (A. ..., B. ..., option definitions in the prompt) followed by the Qwen3 non-thinking assistant header. The logits of the first answer token are restricted to the option-letter tokens (A..Z), divided by a per-type temperature and softmaxed. That distribution is the answer. The weights are an ordinary Qwen3 causal LM; the readout is done by the open-spark-Jev repository (https://github.com/abhishek085/open-spark-jev) code (open_spark_jev.model.MenuScorer, open_spark_jev/serve), which also serves the POST /v1/decide API. calibration.json holds the temperatures, fitted for every question type this release: choice 2.026, score 1.149, noul 1.318.

Results

Own held-out splits (locked test / challenge draw from scenario families never seen in training)

External sets (no training overlap)

JevBench (public tiers only)

Accuracy on Benchmark Heaven's JevBench v1.2, public items only (231 of 534 total; the judge tier and part of each other tier are private and were not run): | tier | n | accuracy | |---|---:|---:| | easy | 48 | 1.000 | | standard | 72 | 0.847 | | hard | 111 | 0.523 |

This is not the official JevBench Score (that also needs the private item set plus Calibration, Speed and Cost axes measured under JevBench's own protocol); it is a same-methodology comparison point across our own releases.

Latency

Batch size 1, idle GPU, one NVIDIA DGX Spark (GB10): | serving stack | precision | p50 | decisions/s | |---|---|---:|---:| | Hugging Face Transformers (in-process) | bf16 | 65.9 ms | 15.1 | | vLLM | bf16 | 61.6 ms | 16.2 | | vLLM | NVFP4, MLP layers only | 37.1 ms | 27.1 |

The HF Transformers bf16 number is carried over from the v3 release on the same backbone and prompt set (LoRA-merged weights don't change inference cost); the vLLM numbers were measured fresh on this checkpoint. NVFP4 quantization (weights of the MLP projections only; attention and the LM head stay bf16, since the option-letter readout depends directly on the LM head's logits) gives a real 1.66x speedup on the DGX Spark's Blackwell tensor cores. Unlike on the 1.7B checkpoint (where the same recipe cost 9-13 accuracy points and was not shipped), the 4B tolerates it much better — see the accuracy table below. The NVFP4 checkpoint is published separately at `abhishek085/spark-s1-4b-v5-nvfp4` (4.0 GB, needs a Blackwell-generation GPU and vLLM/TensorRT-LLM to serve); it can also be reproduced with scripts/quant/ptq_nvfp4.py --cfg NVFP4_MLP_ONLY_CFG in the source repository.

NVFP4 (MLP-only) accuracy, on JevBench's public tiers (the only accuracy check run on this variant so far)

Unlike the 1.7B (where the same MLP-only recipe cost 9-13 points across every external set), the 4B holds its easy/standard-tier accuracy exactly and only gives up 4.6 points on the hard tier. This is the only accuracy check run on the NVFP4 variant so far (JevBench public tiers); it has not yet been scored on the own-splits/external-set battery the bf16 numbers above use, so treat the accuracy cost as smaller than the 1.7B's but not fully characterized outside JevBench.

Limitations

• —Trained and evaluated on Jev-style decisions only for this release; general text classification (topic, sentiment, NLI, etc.) is out of scope and untested here.
• —Boolean/score temperatures are now fitted (unlike v3, where they were left at 1.0) but on a smaller sample for some question types; recalibrate on your own labelled outcomes before relying on thresholds.
• —An RLCD-direct pass was tried on the 1.7B checkpoint (same data/recipe family) on a hard-mined subset of unseen rows and made results worse across the board, not better; it was not applied here either. This release is supervised fine-tuning only.
• —Answers depend on the state and option definitions you give it; it never explains itself. Do not use it as the only gate on high-impact actions.

Reproducibility

Code: github.com/abhishek085/open-spark-jev. Training: configs/train/sft_v5.yaml (python -m open_spark_jev.train.sft --config configs/train/sft_v5.yaml --set model=Qwen/Qwen3-4B output_dir=checkpoints/v5-4b). Evaluation: python -m open_spark_jev.eval.osdg, python -m open_spark_jev.eval.external. Single seed; no confidence intervals reported yet.

License

Apache-2.0 (the Qwen/Qwen3-4B backbone is Apache-2.0). The training data was generated and verified with Gemma-4-26B-A4B, Qwen3.6-35B-A3B, Qwen3.6-27B and NVIDIA-Nemotron-3-Super-120B-A12B (used as a writer for a subset of the harness packs); check those terms for commercial use.