anziank/grio-qwen2.5-1.5b-instruct-coreml-stateful

Qwen 2.5 1.5B Instruct (Stateful KV) — CoreML Stateful KV-Cache (ctx≤512)

On-device CoreML .mlpackage converted from `Qwen/Qwen2.5-1.5B-Instruct` with stateful KV cache (ct.StateType) for efficient token-by-token generation.

Designed for the coremlkv: runner in GrioKit — CoreMLTranslationProvider handles stateful generation via makeState() + prediction(from:using:state:).

Model details

Verified output (greedy, deterministic)

Smoke harness: scripts/coreml/path_b_convert.py --model qwen2.5-1.5b-stateful --smoke-only against the pre-compiled .mlmodelc.

Basic capability (`"The capital of France is"`): "The capital of France is Paris." — 8 decode tokens. Same answer matches PyTorch FP16 reference output.

Translation pack (10 cases, system prompt: `"You are a French translator. Translate the user message to French. Output only the translation."`):

Postprocess pack (8 cases, light-touch transcript copy-edit): all 8 cases ran without graph errors. Headline test sight → site (obvious-homophone) failed (kept sight); idiom/cultural-phrase leave-alone tests passed (Such is life, Life in pink); entity capitalization passed (Bill France Sr. ... NASCAR); the model overreaches on the "light touch" prompt — drops fillers and rephrases (e.g., make sure → ensure, move into → move on to) where the prompt explicitly forbids that. Take this as a prompt-engineering caveat for postprocess use, not a model defect.

Headline takeaway: the model is a competent 1.5B-class French translator and a usable but-imperfect copy-editor at this size. No graph-corruption symptoms; every output is a coherent on-task sentence.

Observed performance

Device: Apple M1 Pro, 16 GB, macOS 26.5 Tahoe. Prompt: "The capital of France is" (34 tokens after chat template).

Subsequent prefills in the same process (warm KV state was reset between cases in the smoke packs) ran 246–665 ms on .cpuAndGPU. The cold prefill numbers are one-time per OS install; CoreML caches the compiled bundles in ~/Library/Caches/com.apple.e5rt.e5bundlecache/ and reuses them across processes.

Translation-pack decode latency on `.cpuAndGPU` (warm): 209–236 ms/tok across all 10 cases — stable, no per-input outliers.

Recommendation: ship with .cpuAndNE for production. The ANE compile cost is paid once, then every subsequent invocation is ~45% faster than .cpuAndGPU. For first-launch UX, consider warming the model in the background before the user needs it.

Runtime gotchas (please read before integrating)

1. 1.Qwen 2.5 uses two stop tokens in `generation_config.json`: <|endoftext|> (id 151643) and <|im_end|> (id 151645). tokenizer.eos_token_id returns only <|im_end|>. Read the full list for robustness. swift-transformers ≥ 1.3 handles this correctly.
2. 2.*`causalMask` is a length signal, not the actual causal mask* — shape [1, 1, 1, end_step], all-zero float16. The stateful wrapper inside the model reads only end_step from its shape and builds the proper (1, 1, q_len, end_step) lower-triangular mask internally. This decision matters when porting to other Swift / Python harnesses: do not try to send a "real" causal mask through the public interface — its query dimension is pinned to 1 in the export and a real mask would either be rejected at runtime or broadcast incorrectly.
3. 3.Reset state between independent generations. Call CoreMLTranslationProvider.shared.resetSession() (or model.resetState() on the stateful LanguageModel) between unrelated requests. Stale KV state from a previous sequence corrupts subsequent generations.
4. 4.`.mlmodelc` is included alongside `.mlpackage`. Use the .mlmodelc path for loading — it skips the multi-minute ANE compilation that happens on first .mlpackage load.

Conversion notes (for the CoreML community)

1. 1.`attn_implementation="sdpa"` is mandatory for Qwen 2.5 1.5B at FP16. The HF default "eager" triggers optimize_repeat_ops.py:433 RuntimeWarning: overflow encountered in cast during MIL conversion and causes Qwen's attention to overflow to NaN in PyTorch CPU reference checks. SDPA avoids both.
2. 2.`run_decompositions({})` SIGSEGVs at 1.5B+ during MIL conversion. Use torch.export.default_decompositions() instead.
3. 3.`create_causal_mask` in `transformers` ≥ 5 returns 4-D masks unchanged. Sending the public [1, 1, 1, end_step] length-signal mask straight into attention layers therefore broadcasts across the query axis and gives BIDIRECTIONAL attention during batch prefill. Llama 3.x tolerates this (4:1 GQA, 8 KV heads). Qwen 2.5 1.5B (6:1 GQA, only 2 KV heads) does not — it collapses to !-token spam. This artifact's wrapper builds the real (1, 1, q_len, end_step) lower-triangular mask internally from (k_idx − q_idx) ≤ past_seen so attention is causal regardless of what the caller sends.
4. 4.Qwen 2.5 1.5B has FP16-overflowing K activations (|K| ≈ 318 → Q@K^T ≈ 255 196 > 65 504). The wrapper runs SDPA in an FP32 island and casts back to FP16. The FP32 ops are preserved by coremltools and scheduled by ANE on its dedicated FP32 path. The same wrapper is used for Llama 3.x stateful conversions where it is a no-op safety net.
5. 5.`logits` shape metadata may be empty in the .mlpackage description for dynamic outputs. Verify rank with a real predict() call.
6. 6.FP16 greedy decoding is not byte-deterministic across backends. Semantically equivalent to PyTorch FP16 reference for every prompt we have tested.

Usage (Swift — GrioKit coremlkv: runner)

import GrioKit

// Load via CoreMLTranslationProvider using the coremlkv: prefix
let provider = CoreMLTranslationProvider.shared
try await provider.configure(modelID: "coremlkv:qwen2.5-1.5b-stateful", computeUnits: .cpuAndNeuralEngine)

// Translate a sentence (stateful: KV cache persists across tokens)
let result = try await provider.translate(
    "The quick brown fox jumps over the lazy dog.",
    sourceLanguage: "English",
    targetLanguage: "French"
)
print(result) // e.g. "Le rapide renard brun saute par-dessus le chien paresseux."

// Reset KV state between unrelated requests
await provider.resetSession()

Keep tokenizer.json, tokenizer_config.json, config.json, generation_config.json (and chat_template.jinja for Gemma) as siblings of the .mlpackage on disk. All are bundled in this repo.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1 via path_b_convert.py --model qwen2.5-1.5b-stateful --stateful.

Stateful KV weights are ct.StateType buffers (keyCache, valueCache), shaped (num_layers, 1, num_kv_heads, 512, head_dim). Post-convert quantization (--quantize int8) applies linear_quantize_weights with linear_symmetric mode to the saved .mlpackage; the state buffers are runtime tensors and are unaffected by weight quantization.

License

Apache-2.0. Weights from Qwen/Qwen2.5-1.5B-Instruct by Qwen Team / Alibaba Cloud. Re-uploaded as a CoreML port; original model card terms apply.