anziank/grio-qwen2.5-1.5b-instruct-coreml-stateful-int8

Qwen 2.5 1.5B Instruct (INT8, Stateful KV) — CoreML Stateful KV-Cache (ctx≤512)

INT8-weight-quantized sibling of the FP16 stateful Qwen 2.5 1.5B artifact. Weights stored at 8 bits with per-channel symmetric linear quantization; activations stay FP16 (W8A16). The KV-cache state inputs (keyCache, valueCache) are preserved unchanged, so the same coremlkv: runner in GrioKit loads this artifact with no code change.

CoreMLTranslationProvider in GrioKit handles stateful generation via makeState() + prediction(from:using:state:).

Model details

Related variants

For the 3B sibling family, post-training INT8 / INT4 weight quantization does not survive on the deeper Qwen 2.5 3B Instruct base — only the FP16 3B artifact is viable. This 1.5B INT8 build is the only quantized stateful Qwen variant that holds quality.

Verified output (greedy, deterministic)

Smoke harness: scripts/coreml/path_b_convert.py --model qwen2.5-1.5b-stateful --quantize int8 --smoke-only against the pre-compiled .mlmodelc.

Basic capability (`"The capital of France is"`): "The capital of France is Paris." — 8 decode tokens. Matches FP16 sibling output.

Translation pack (10 cases, system prompt: `"You are a French translator. Translate the user message to French. Output only the translation."`): all 10 cases produced coherent French output. No graph errors, no truncation, no language mixing.

Post-processing pack (8 cases, light-touch transcript copy-edit): 6 of 8 cases clean.

The proper-noun corruption on row 4 (Bill France Sr. → BILLY FRANCE S.) is the only output that differs from the FP16 sibling on this pack. It is the classic INT8 per-channel failure mode: rare-token rows in the lm_head share a quantization scale with high-magnitude bulk rows, losing precision on entity-formatting argmaxes.

Headline takeaway: translation quality matches the FP16 sibling closely; post-processing is usable with one specific regression on rare proper nouns. For strict entity formatting in post-processing, prefer the FP16 sibling.

Observed performance

Device: Apple M1 Pro, 16 GB, macOS 26.5 Tahoe. Compute units: .cpuAndGPU.

Translation-pack decode latency on .cpuAndGPU: 208–212 ms/tok across rows 2–10 (warm); row 1 hits 308 ms/tok as a one-time cold-start tax (first inference after model load). Post-pack decode latency: 210–213 ms/tok across all 8 cases, stable. No per-input outliers once warm.

INT8 weight quantization does not materially change throughput vs FP16 at this model size — the bottleneck is activation compute, not weight bandwidth. The win is disk + RSS (~½ each), not speed. .cpuAndNE was not benchmarked for this INT8 variant; the FP16 sibling shows .cpuAndNE is ~1.8× faster at decode, but INT8 quantization sometimes interacts differently with ANE-specific code paths so we make no promise here without measurement.

Runtime gotchas (please read before integrating)

1. 1.Same stop-token list as the FP16 sibling. <|endoftext|> (151643) and <|im_end|> (151645). tokenizer.eos_token_id returns only <|im_end|>; read both from generation_config.json.
2. 2.`causalMask` is a length signal, not the actual causal mask. Shape [1, 1, 1, end_step], all-zero float16. The stateful wrapper builds the real (1, 1, q_len, end_step) lower-triangular mask internally. See the FP16 README for the rationale.
3. 3.Reset state between independent generations. Call CoreMLTranslationProvider.shared.resetSession() (or model.resetState()) between unrelated requests.
4. 4.Avoid `CPU_ONLY`. CoreML routes FP16 transformer activations through BNNS in CPU_ONLY mode, which produces NaN for this artifact class. Use .cpuAndGPU or test .cpuAndNeuralEngine.
5. 5.Cold ANE compile. First load builds the .mlmodelc from the .mlpackage (~85 s). Ship the .mlmodelc alongside the .mlpackage (included in this repo) to skip this on first launch.

Quantization recipe

Produced from the FP16 master via the scripts/coreml/quantize_stateful_int8.py helper in the Grio repo:

mamba activate claude-grio
python scripts/coreml/quantize_stateful_int8.py \
    --input  Qwen2.5-1.5B-Instruct-stateful.mlpackage \
    --output Qwen2.5-1.5B-Instruct-stateful-int8.mlpackage
# defaults: mode=linear_symmetric, granularity=per_channel, dtype=int8, weight_threshold=512

The script verifies that keyCache and valueCache state declarations survive the quantization pass before saving. 342 ops quantized in ~30 s; overall wall time including coremltools load/save round-trip ~13 min.

Conversion provenance (inherited from FP16 master)

The FP16 source was produced by path_b_convert.py --model qwen2.5-1.5b-stateful --stateful with the documented FP32-SDPA-island fix for Qwen's overflowing K activations (|K| ≈ 318). All structural conversion notes from the FP16 README apply to this artifact (SDPA mandatory at FP16, default_decompositions() to avoid SIGSEGV at 1.5B+, internal causal-mask construction). INT8 weight quantization is applied as a post-training pass on the converted artifact; it does not re-run the conversion graph.

Usage (Swift — GrioKit coremlkv: runner)

Real signature of CoreMLTranslationProvider.translate(...) from GrioKit/Sources/GrioKit/CoreMLTranslationProvider.swift:

import GrioKit
import CoreML

let provider = CoreMLTranslationProvider.shared

// Build the chat-templated prompt for the family (Qwen 2 here).
// In production this comes from AnyLMTranslationProvider.PromptStyle.coreMLChatPrompt(...)
let prompt = """
<|im_start|>system
You are a French translator. Translate the user message to French. Output only the translation.<|im_end|>
<|im_start|>user
The quick brown fox jumps over the lazy dog.<|im_end|>
<|im_start|>assistant

"""

// Locate the .mlpackage on disk; `tokenizerFolder` defaults to its parent.
let modelURL = URL(fileURLWithPath: "/path/to/Qwen2.5-1.5B-Instruct-stateful-int8.mlpackage")

let result = try await provider.translate(
    prompt,
    modelURL: modelURL,
    maxNewTokens: 128,
    computeUnits: .cpuAndGPU,
    tokenizerFolder: modelURL.deletingLastPathComponent()
    // Optional `sampling: SamplingOverrides(...)` for temperature / topK / repetitionPenalty
)
print(result) // e.g. "Le renard brun rapide saute par-dessus le chien paresseux."

// Reset KV state between unrelated requests
await provider.resetSession()

In the Grio app this is wired through TranslationCoordinator (which builds the chat-template prompt and routes via the coremlkv: model-key prefix). For post-processing, CoreMLKVPostProcessingEngine uses the sibling provider.process(chatPrompt:modelURL:...) entry point with post-friendly sampling defaults.

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json as siblings of the .mlpackage on disk — tokenizerFolder defaults to modelURL.deletingLastPathComponent(). All four files are bundled in this repo.

License

Apache-2.0. Weights from `Qwen/Qwen2.5-1.5B-Instruct` by Qwen Team / Alibaba Cloud. INT8 weight quantization performed locally via coremltools; redistribution under the same Apache-2.0 license terms as the base model.