VladHong/Ling-3.0-Tiny-APEQ-SubMini

Ling-3.0-Tiny APEQ SubMini (GGUF)

This repository contains one custom GGUF quantization of inclusionAI/Ling-3.0-tiny, made for local llama.cpp inference where storage and decode speed matter.

It is an APEQ-style, per-tensor mixed-precision profile. It is not a standard uniform Q5K quantization: the sparse MoE FFN uses Q2K, most unchanged tensors use Q6K/Q80/F32, and selected later attention tensors use Q5_K.

File

The file is 6,897,472 bytes (0.225%) smaller than the local Mini comparator used during this experiment.

Quick start — llama.cpp

Use a recent llama.cpp build with support for the Ling / bailingmoe3 architecture.

llama-cli \
  -m Ling-3.0-Tiny-APEQ-SubMini-3.06GB.gguf \
  -ngl 99 \
  -cnv \
  --reasoning off

To start an OpenAI-compatible local server:

llama-server \
  -m Ling-3.0-Tiny-APEQ-SubMini-3.06GB.gguf \
  -ngl 99 \
  -sm none \
  -c 4096 \
  --reasoning off

The base model's native chat template is embedded in the GGUF. The original model recommends temperature=1.0, top_p=0.95, and top_k=20; use deterministic settings only when reproducing the comparison below.

Validation

This is a local comparative result, not a claim of universal quality equivalence.

The final artifact was regenerated twice from BF16 with the same exact tensor map and had the same SHA-256 each time. On a fresh, disjoint 24-case deterministic ZxBench-derived holdout, it matched the Mini comparator exactly:

Throughput was measured with llama.cpp build b10516 on a Tesla V100-SXM2-16GB, full GPU offload, no speculative decoding, and three samples of a 512-token prefill / 128-token decode workload. Hardware, context length, batching, runtime build, and sampling parameters will affect results.

The holdout used deterministic local graders adapted from ZxBench scenarios. It is a representative comparative screen, not a full hosted ZxBench run with optional AI judges or sandboxes.

Quantization profile

The profile preserves higher precision in the early sensitive attention path and lowers standard attention tensors in blocks 5–23, plus blk.4.attn_q.weight, to Q5_K. Broader reductions were screened and rejected due to measurable regressions.

License and attribution

This quantization is derived from inclusionAI/Ling-3.0-tiny and follows its MIT license. Please review the upstream model card for the original model's capabilities, limitations, and usage guidance.