soyaakinohara/Spark-X2.5-4B-Heretic-jp-gguf

Spark-X2.5-4B Heretic JP GGUF

GGUF quants of a Japanese-adapted, refusal-attenuated derivative of the XHToken/Spark-X2.5-4B model.

The pipeline is: Heretic refusal attenuation → Japanese continued pretraining → Japanese instruction tuning → tool-calling repair. This release includes all four stages.

Files

Spark-X2.5-4B-Heretic-jp-BF16.gguf
Spark-X2.5-4B-Heretic-jp-Q8_0.gguf
Spark-X2.5-4B-Heretic-jp-Q6_K.gguf
Spark-X2.5-4B-Heretic-jp-Q5_K_M.gguf
Spark-X2.5-4B-Heretic-jp-Q4_K_M.gguf

llama.cpp usage

A Spark-compatible llama.cpp fork is required (upstream does not support this architecture). Use XHToken/llama.cpp. Single-turn CLI example:

llama-cli -m Spark-X2.5-4B-Heretic-jp-Q8_0.gguf -n 512 -t 16 -ngl 99 \
  --no-conversation -st -p "日本の首都は?"

Server example:

llama-server -m Spark-X2.5-4B-Heretic-jp-Q8_0.gguf -c 8192 -t 16 -ngl 99 \
  --host 127.0.0.1 --port 8899

Notes:

• —This is a thinking model. Responses start with [Start thinking] and end thinking with [End thinking] before the answer. Use max_tokens of 1024 or more; a 200-token budget gets consumed by thinking alone.
• —For tool use, pass a tools array to /v1/chat/completions. tool_choice: "auto" fires correctly on these weights.

For a network-exposed server, add your own authentication, firewall, and access controls before exposing it beyond a trusted LAN.

Reported local performance

The model was prepared and tested on:

OS:       Ubuntu 24.04
GPUs:     NVIDIA GeForce RTX 5060 Ti 16GB x 2
RAM:      32GB + 23GB swap
Runtime:  llama.cpp CUDA build (Spark fork)

On this machine, Q80 generation reached a reported **75.9 tokens/second** and Q4K_M reached 105.9 tokens/second. Actual speed depends on context length, prompt length, sampling settings, CUDA/llama.cpp version, and background workload.

Provenance

Base model

• —XHToken/Spark-X2.5-4B
• —Base revision used locally: 1e4c2477f330d1f11304d0dc092a9dbf4075f470
• —License: Apache-2.0

Adaptation steps

1. 1.Refusal attenuation with Heretic (refusals 58→3/100, KL 0.0118).
2. 2.Japanese continued pretraining on 50k Japanese Wikipedia articles (train loss 2.77→2.20).
3. 3.Japanese instruction tuning on 50k rows (train loss 2.59→1.50).
4. 4.Tool-calling repair on 8k tool-use rows (train loss 1.48→0.57). After this step, complex tool definitions fire correctly (function-name match 96% on a held-out Japanese tool-call set).

Training ran in BF16 + LoRA (r16) with a 2-GPU layer split. Validation: Q4KM perplexity 6.75 on held-out Japanese data; think-to-answer completion verified on Q4KM.

Datasets

• —fn-aka-mur/wiki40b_ja (continued pretraining)
• —izumi-lab/llm-japanese-dataset-vanilla (instruction tuning, CC-BY family)
• —nappa0326/glaive-function-calling-v2-sharegpt-japanese (tool repair, MIT)
• —Tonari-no-usagi/Japanese_Function_Calling_Dataset (evaluation only; CC-BY-NC, not used for training)

The local conversion and quantization used a Spark-compatible llama.cpp fork (XHToken/llama.cpp, build 10512, commit a698f1cc3).

AI assistance disclosure

The local model preparation workflow, training, conversion, quantization, validation, and this model card were performed with assistance from Hermes Agent. The model was then reviewed and published by the repository owner.

Responsible use

This is an uncensored / refusal-removed model. It may produce content that an aligned model would refuse, including unsafe, illegal, or harmful material. It has no reliable built-in safety layer. Use appropriate access controls, moderation, logging, and human review for any deployment, and comply with all applicable laws and policies.

The model is provided as-is. Users are responsible for prompts, outputs, and any downstream actions based on them.

日本語

概要

XHToken/Spark-X2.5-4B をベースに、無検閲化と日本語適応を行ったモデルの GGUF です。

無検閲化には Heretic による refusal 減衰を使い、 日本語 Wikipedia での継続事前学習と日本語指示データでの調整を行っています。 ツール呼び出しの追加学習も行い、 複雑なツール定義でも正規の tool call が出ることを確認しています。

使い方

llama.cpp の Spark 対応フォークが必要です。

llama-cli -m Spark-X2.5-4B-Heretic-jp-Q8_0.gguf -n 512 -t 16 -ngl 99 \
  --no-conversation -st -p "日本の首都は?"

thinking 系モデルのため、応答は [Start thinking] から始まります。 max_tokens は 1024 以上を推奨します。