sriq-ai/sriq-spark-v1.4

<img src="./banner.png" width="100%" alt="SRIQ-Spark-V1.4 banner">

SRIQ-Spark-V1.4

SRIQ-Spark-V1.4 is a language model by SRIQ, supervised fine-tuned from XHToken/Spark-X2.5-4B. Our focus is reasoning traces that verify their own answer before committing to it.

The name combines SRIQ and Spark.

Training

Supervised fine-tuning on sriq-ai/sriq-sft-v1.4, a ShareGPT-format set whose assistant targets pair compressed Simplified Chinese reasoning with the original final answer. The compression step never rewrites the final answer.

v1.4's defining change over v1.3 is a final verification step: traces close by checking the answer against the original constraints — substituting the result back, re-deriving the invariant, walking the stated edge cases — introduced by 验： and often ending on an explicit judgement such as 答对。 or 故对。 This costs roughly 22% longer traces than v1.3; the goal is a checked answer, not a shorter one.

Training used Unsloth with a LoRA adapter over the bfloat16 base checkpoint, on a single NVIDIA RTX 5090.

Hyperparameters

Target modules follow this architecture's own layer names, not the Llama-style defaults — Spark-X2.5 uses a fused QKV projection and calls its attention output out_proj:

q_k_v_proj  out_proj  gate_proj  up_proj  down_proj

The adapter is merged into the weights published here, so no PEFT adapter is needed at inference time — load the repo directly.

Training coverage

30 steps at an effective batch size of 8 is 240 sequences against a dataset of 825 rows — about 0.29 of one epoch. This is a short run. Treat it as a first pass, not a converged model.

Evaluation

No benchmarks were run for this release. SRIQ-Spark-V1.4 ships unmeasured: the claims above describe the training setup, not verified gains in accuracy, verification rate, or answer quality. Evaluate it on your own workload before relying on it.

Compatibility

Spark-X2.5 is a custom architecture that ships its own modelling code, so trust_remote_code=True is required. This repo ships a patched copy of that code, because the upstream version does not run under current Transformers. Two changes, both in modeling_spark.py:

1. 1.`_tied_weights_keys` is a dict, not a list. Upstream declares _tied_weights_keys = ["lm_head.weight"], the Transformers 4.x convention. Transformers 5.x calls .keys() on it during post_init(), so loading the upstream model fails with AttributeError: 'list' object has no attribute 'keys'. Here it is {"lm_head.weight": "model.embedding.weight"}.

1. 1.Activations are no longer cast to the raw weight dtype. Spark2_5DecoderLayer.forward cast hidden states with .to(self.mlp.gate_proj.weight.dtype). Under bitsandbytes 4-bit that weight is packed uint8, so activations became uint8 and attention died with NotImplementedError: "baddbmm_cuda" not implemented for 'Byte'. A helper now resolves the quantization compute dtype instead. This has no effect at bfloat16, but makes 4-bit loading of this repo work.

Because of change 1, the shipped code targets Transformers 5.x. Neither change touches the maths of the forward pass at bfloat16.

One further caveat, not patched: this model's remote code builds its RoPE cos/sin cache once on the device of the first layer and reuses it across all layers, so it cannot be sharded across multiple GPUs by device_map="auto" — you get Expected all tensors to be on the same device. Pin it to one GPU. At 4B parameters in bfloat16 it fits comfortably on a single 32 GB card.

vLLM does not recognise the spark2_5 architecture and will not serve this repo without a custom model plugin.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sriq-ai/sriq-spark-v1.4"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, trust_remote_code=True, device_map={"": 0}
)

messages = [{"role": "user", "content": "What is 17 * 23? Answer briefly."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Reasoning is emitted inside <think> tags in Simplified Chinese, closing with the 验： verification block, followed by the final answer. Output from the snippet above:

题：17*23。算：17*20=340，17*3=51，340+51=391。验：23*17=391。答：391。</think>391

Credits

Developed by SRIQ. Base model by the XHToken team.