VNPen/vnpen-writer-2b-v0.1-preview-GGUF

vnpen-writer-2b-v0.1-preview-GGUF

GGUF / Ollama builds of `VNPen/vnpen-writer-2b-v0.1-preview`, for on-device deployment.

[English](#english) · [中文](#中文)

⚠️ These are quantisations of a preview checkpoint, not of a v0.1 release. Everything the source card says about quality, scope and known problems applies here unchanged. Read it first. ⚠️ 这些是 preview 权重的量化版本，不是 v0.1 正式发布。 源模型卡中关于质量、适用范围与已知问题的说明在此完全适用，请先阅读。

<a name="english"></a>

English

Files

Perplexity is measured on a held-out slice of the project's own corpus that is disjoint from the importance-matrix calibration set — 300 samples / 336,872 tokens, rendered through the model's own chat template. The quant loss below is therefore measured, not memorised.

RAM figures are the model file plus an 8,192-token context and a little runtime overhead — roughly what you need free to run it.

Which one

Take `Q4_K_M`. It gives up 1.84% perplexity for a file 66% smaller than BF16, and it is the only build here whose size/quality trade is not beaten by a neighbour.

IQ4_XS is included for completeness but is usually the wrong pick: it saves only 6% over Q4_K_M and pays 3.07% perplexity for it. The reason is that this model ties its input and output embeddings, and token_embd.weight (248,320 × 2,048 — 27% of all parameters) is held at Q6K in every K-quant below Q6K. That one tensor dominates the file, so the low-bit formats have little left to shrink.

Run it with Ollama

Use the `Modelfile`. It is what carries the tuned defaults below, and this model needs them:

hf download VNPen/vnpen-writer-2b-v0.1-preview-GGUF \
  writer-2b-preview-Q4_K_M.gguf Modelfile --local-dir vnpen
cd vnpen && ollama create vnpen-writer-2b -f Modelfile
ollama run vnpen-writer-2b "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。"

The one-liner works too:

ollama run hf.co/VNPen/vnpen-writer-2b-v0.1-preview-GGUF:Q4_K_M

but it does not pick up this repo's `Modelfile` — Ollama builds that model from the GGUF alone, so it gets the embedded chat template and the stop token but not the repetition settings. Set them in the session yourself, or the degeneration described below will show up:

/set parameter repeat_penalty 1.1
/set parameter repeat_last_n 256

Run it with llama.cpp

llama-cli -m writer-2b-preview-Q4_K_M.gguf --jinja \
  -p "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。" \
  --temp 0.8 --top-p 0.95 --repeat-penalty 1.1 --repeat-last-n 256 -c 8192

The chat template is embedded in the GGUF, so --jinja is enough.

The defaults are not the stock defaults

The shipped Modelfile deliberately overrides Ollama's defaults:

With these defaults a 31-line scene came back with 31/31 lines in the speaker：text format, 31 unique lines, and done_reason: stop — the repetition the source card warns about did not appear.

Context is cheap on this architecture

Qwen3.5 is a hybrid. Of the 24 layers only 6 run full attention (layers 3, 7, 11, 15, 19, 23); the other 18 are linear-attention (gated delta-net) layers whose recurrent state is a fixed size regardless of sequence length.

So the KV cache costs about 12 KB/token instead of the ~48 KB/token a dense 24-layer model of this shape would need:

Raise num_ctx freely — it is far cheaper here than the parameter count suggests. The model declares 262,144.

Speed

llama-bench, Q4KM, llama.cpp build 60081bb:

The CPU row is a server CPU restricted to 8 threads, not a phone and not a Raspberry Pi — read it as an order of magnitude, not as a promise about any particular device. 30 t/s is comfortably faster than reading speed.

How these were built

• —Converted with llama.cpp build 60081bb, convert_hf_to_gguf.py, architecture qwen35.
• —`--no-mtp` is required for this checkpoint. The base Qwen/Qwen3.5-2B carries 15 mtp.* multi-token-prediction tensors and config.json keeps mtp_num_hidden_layers: 1. The merged release is the text decoder only and does not carry them, so a default conversion writes block_count = 25 against 24 blocks of weights and the result fails to load with check_tensor_dims: tensor 'blk.24.attn_norm.weight' not found. --no-mtp writes block_count = 24 and drops the NextN metadata. MTP only serves speculative decoding, which is not used here.
• —The importance matrix was computed on the project's own training corpus, not on generic English text: 1,200 samples / 1,373,121 tokens drawn from train_writer_preview.jsonl (117,077 rows) by seeded reservoir sampling and rendered through the model's own chat template, at -c 2048. Calibrating on the distribution the model is actually deployed into is why the 4-bit loss stays under 2%. writer-2b-imatrix.gguf is included so you can requantise.
• —The 133 norm and SSM parameters (A_log, dt_bias, ssm_norm, conv1d) are kept at F32 in every build, which is what a hybrid model needs.

Provenance

Quantised from VNPen/vnpen-writer-2b-v0.1-preview, which is itself built from Qwen/Qwen3.5-2B (Copyright the Qwen team, Alibaba Group, Apache-2.0) by LoRA supervised fine-tuning, merged, with the vision tower never loaded. Apache-2.0 applies to these files too. The visual-novel corpus is not redistributed.

<a name="中文"></a>

中文

文件

困惑度在与 imatrix 校准集不相交的留出集上测量——300 条样本 / 336,872 token，按模型自身的 chat template 渲染。因此下表的量化损失是实测的， 而不是在校准数据上的自我复现。

内存一列是模型文件加 8,192 token 上下文和少量运行时开销，约等于运行所需的空闲内存。

选哪个

选 `Q4_K_M`。 相比 BF16 体积小 66%，代价只有 1.84% 的困惑度， 是这组里唯一没有被相邻档位比下去的取舍点。

IQ4_XS 为完整性而提供，但通常是错误选择：它只比 Q4_K_M 小 6%， 却多付 3.07% 的困惑度。原因是本模型输入输出嵌入共享权重， token_embd.weight（248,320 × 2,048，占全部参数的 27%）在所有低于 Q6K 的 K-quant 里都被保留为 Q6K。这一个张量主导了文件体积，低比特格式已无多少可压。

用 Ollama 运行

请用 `Modelfile`。 下述调好的默认参数是靠它带上的，而这个模型需要这些参数：

hf download VNPen/vnpen-writer-2b-v0.1-preview-GGUF \
  writer-2b-preview-Q4_K_M.gguf Modelfile --local-dir vnpen
cd vnpen && ollama create vnpen-writer-2b -f Modelfile
ollama run vnpen-writer-2b "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。"

一行命令也可以：

ollama run hf.co/VNPen/vnpen-writer-2b-v0.1-preview-GGUF:Q4_K_M

但这条命令不会读取本仓库的 `Modelfile`——Ollama 只从 GGUF 构建模型， 因此它能拿到内嵌的 chat template 和停止词，却拿不到重复惩罚设置。 请在会话里自行设置，否则下文描述的退化会出现：

/set parameter repeat_penalty 1.1
/set parameter repeat_last_n 256

用 llama.cpp 运行

llama-cli -m writer-2b-preview-Q4_K_M.gguf --jinja \
  -p "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。" \
  --temp 0.8 --top-p 0.95 --repeat-penalty 1.1 --repeat-last-n 256 -c 8192

chat template 已嵌入 GGUF，加 --jinja 即可。

默认参数是特意改过的

随附的 Modelfile 有意覆盖了 Ollama 的出厂默认：

用这组默认参数实测：一个 31 行的场景返回 31/31 行符合「说话人：文本」格式、 31 行全部唯一、done_reason: stop——源模型卡警告的重复退化没有出现。

这个架构上，上下文很便宜

Qwen3.5 是混合架构：24 层中只有 6 层是全注意力（第 3、7、11、15、19、23 层），其余 18 层是线性注意力（gated delta-net），其循环状态是与序列长度无关的 固定大小。

因此 KV cache 约为 12 KB/token，而不是同规模稠密 24 层模型所需的约 48 KB/token：

可以放心调大 num_ctx，它比参数量给人的直觉便宜得多。模型声明支持 262,144。

速度

llama-bench，Q4KM，llama.cpp 构建 60081bb：

CPU 一行是限制到 8 线程的服务器 CPU，不是手机也不是树莓派——只能当数量级参考， 不构成对任何具体设备的承诺。30 t/s 已经明显快过阅读速度。

构建方式

• —使用 llama.cpp 构建 60081bb 的 convert_hf_to_gguf.py 转换，架构 qwen35。
• —本权重必须加 `--no-mtp`。 基座 Qwen/Qwen3.5-2B 带有 15 个 mtp.* 多 token 预测张量，且 config.json 保留了 mtp_num_hidden_layers: 1。 合并发布的权重只有文本解码器、不含这些张量，因此默认转换会写入 block_count = 25 而实际只有 24 块权重，加载时报 check_tensor_dims: tensor 'blk.24.attn_norm.weight' not found。 --no-mtp 会写入 block_count = 24 并移除 NextN 元数据。MTP 只服务于 投机解码，此处并不使用。
• —重要性矩阵在项目自己的训练语料上计算，而非通用英文文本：从 train_writer_preview.jsonl（117,077 行）以定种子蓄水池采样抽取 1,200 条 / 1,373,121 token，按模型自身 chat template 渲染，-c 2048。 在模型实际部署的同一分布上做校准，正是 4-bit 损失能压在 2% 以内的原因。 仓库内附 writer-2b-imatrix.gguf，可用于自行重新量化。
• —133 个 norm 与 SSM 参数（A_log、dt_bias、ssm_norm、conv1d）在所有 版本中均保持 F32，这是混合架构所必需的。

来源与许可

量化自 VNPen/vnpen-writer-2b-v0.1-preview；后者由 Qwen/Qwen3.5-2B （版权归 Qwen 团队 / 阿里巴巴集团，Apache-2.0）经 LoRA 监督微调并合并而来， 视觉塔从未被加载。本仓库文件同样适用 Apache-2.0。视觉小说语料不随之分发。