mlboydaisuke/qwen3.5-0.8B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->  Measured decode — iPhone 17 Pro: 72 tok/s · Mac (M4 Max): 210 tok/s (DeviceMark row qwen3.5-0.8b, int8hu bundle · data) <!-- gen-cards:devicemark end -->

Qwen3.5-0.8B — Apple Core AI (.aimodel)

[!NOTE] Update 2026-07-15: gpu-pipelined-b2/ adds qwen3_5_0_8b_decode_int8hu_block32_sym re-exported with coreai-core 1.0.0b2, loadable on the OS 27 beta 3 toolchain (June-era b1 bundles fail to load there with a versioned-IR error). The original b1 tree is retained unchanged so existing apps and pinned catalogs keep working. The b2 decode bundle is the exact artifact measured on DeviceMark.

Qwen3.5-0.8B converted to Apple's Core AI (the Core ML successor announced at WWDC26), ready to run on iOS 27 / macOS 27. A hybrid linear-attention model — 3 gated-delta (Mamba-style) layers per full-attention layer — running through Core AI's runtime, greedy top-1 exact vs the Hugging Face reference.

This repo publishes one bundle per platform × compute-unit: the best verified configuration (plus the cross-platform gpu-pipelined/ bundle) — each file is the exact artifact behind the published numbers, nothing experimental.

Requires iOS 27 / macOS 27 (Core AI ships with the OS). Conversion code, knowledge base, and the Swift runner: [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo).

<!-- gen-cards:use-it begin id=qwen3.5-0.8b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

Measured decode — iPhone 17 Pro: 72 tok/s · Mac (M4 Max): 210 tok/s (DeviceMark row qwen3.5-0.8b, int8hu bundle · data)

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("qwen3.5-0.8b"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Qwen3.5 0.8B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model qwen3.5-0.8b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "qwen3.5-0.8b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is `Examples/ChatDemo/Sources/QuickStart.swift` — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• —SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• —Info.plist: none needed
• —Entitlements: none needed
• —First run downloads the model — 1.3 GB (Mac) / 1.3 GB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• —Measure in Release — Debug is ~3× slower on per-token host work <!-- gen-cards:use-it end -->

Pick your platform (measured: iPhone 17 Pro / M4 Max, greedy, top-1 exact vs HF)

• —The ★★★ ship bundle adds an untied lmhead quantized as **per-block-32 absmax int8** (`int8hu --head-sym`): the fp16 head was 54% of the per-token weight read on the bandwidth-bound phone — quantizing it is +40% on iPhone (and +3% on M4 Max). Quantize big-vocab heads with plain absmax `symmetric`; the default `symmetricwithclipping` clips outlier head rows and corrupts top-1s. Greedy rollouts are token-identical to the ★★ bundle; same run contract. **Naming note (2026-06-11):** the directory is named `perchan_sym, but its head is per-block-32 — the export script of the day parsed the granularity flag without applying it (since fixed). The numbers above were measured on exactly these bytes and stand. Genuinely per-channel (axis-0) int8 weights turned out to be **broken on the OS 27 beta GPU delegate** (garbage logits — delegate lowering bug, minimal repro in the zoo), so per-block-32 + symmetric` IS the correct ship shape, not a stand-in. The dir name is kept to avoid breaking download paths.
• —The ★★ pipelined bundle is the fastest decode on BOTH platforms, with zero custom kernels: a decode-only loop-free graph (static [1,1] query, dynamic KV) that rides Apple's coreai-pipelined engine (CoreAILanguageModels / EngineFactory — async non-blocking encode, on-GPU argmax sampling, on-device KV growth) instead of a per-token run loop. Token-for-token == the fp16-GPU sequence; 16/16 single-step top-1 vs the fp32 HF oracle. It needs two things from the zoo: the engine extra-states patch (the stock engine carries exactly 2 states; the SSM conv/rec states ride as fixed-shape extras) and COREAI_CHUNK_THRESHOLD=1 at run time (prefill = pipelined S=1 steps ≈ decode speed — so for LONG prompts the ★ static pair below still wins time-to-first-token). Export: conversion/export_qwen3_5_decode_pipelined.py.
• —The ★ int8 fused-kernel monolith is the custom-kernel static config (~3× dynamic, ~1.6× the fp16 static path; the current app-release config): the device GPU is weight-bandwidth-bound, so fused dequant-in-matvec Metal kernels (embedded in the .aimodel — 100% Core AI, WWDC26 session 325) halve the per-token weight stream; the 248320-token tied head runs as a fused matvec + two-level GPU argmax (greedy). Pair it with the prefill companion: the prompt is consumed 16 tokens per pass (in-graph unrolled SSM scan, fp32 recurrence; full blocks only, remainder + generation on the decode graph). Decode output is byte-identical with and without it.
• —The static monoliths are GPU-only — the fp32-SSM form does not produce correct output on the ANE on the OS 27 betas (and custom Metal kernels are GPU-only); use the ANE bundle there.
• —The dynamic int8 bundle (one graph, prefill+decode, 4 states keyCache/valueCache/convState/recState) is the proven Neural-Engine path — and the same file is the best macOS config (the ios-ane/ and macos/ files are identical content; pick by folder for clarity).
• —The SSM while_loop doesn't lower on device delegates — these bundles use the loop-free single-step decode (bit-identical at query_len=1; the prefill graph unrolls the same scan 16× with the state held fp32). Story + gotchas: knowledge base.
• —int8 in the static/dynamic bundles is k-means palettization, gated 8/8 vs the HF oracle in PyTorch before export; the pipelined bundle uses linear per-block int8 (scale-multiply dequant — 256-entry k-means LUTs are slow on the GPU delegate, 204 vs 113 tok/s on M4 Max). int4 does not survive on this model (head/MLP/SSM all degrade; k-means g8 with int8 rescue layers also fails the oracle gate; unlike Gemma 4).

Run it (pipelined, Swift, macOS 27)

git clone https://github.com/apple/coreai-models && cd coreai-models
git apply <(curl -sL https://github.com/john-rocky/coreai-model-zoo/raw/main/apps/coreai-pipelined-extra-states.patch)
COREAI_CHUNK_THRESHOLD=1 swift run -c release llm-benchmark \
  --model <path-to>/gpu-pipelined/qwen3_5_0_8b_decode_int8lin -p 128 -g 256 -n 3

In an app, load the bundle via LanguageBundle + EngineFactory.createEngine (set COREAI_CHUNK_THRESHOLD=1 before engine creation; never call warmup() — it warms shape 256, the S=1 graph rejects it; a 1-token generate is the warmup).

Run it (Python, macOS 27)

import coreai.runtime as rt
model = await rt.AIModel.load(Path("qwen3_5_0_8b_decode_int8.aimodel"),
        rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu()))
fn = model.load_function("main")
out = await fn({"input_ids": rt.NDArray(ids), "position_ids": rt.NDArray(pos)}, state=state)

On device, push the bundle into your app sandbox (xcrun devicectl device copy to --domain-type appDataContainer) — see the Swift runtime notes. Tokenizer: use the original Qwen/Qwen3.5-0.8B tokenizer (swift-transformers loads it directly).

Parity

Greedy decode matches the HF eager reference 8/8 tokens, top-1 exact (prompt-level cosine 0.9999+), verified on macOS conversion and re-verified end-to-end on the iPhone per compute unit. The int8-kernel monolith additionally passes the chained Mac-GPU greedy 8/8 vs the oracle, and the prefill companion passes both an oracle gate and a chunked-vs-q=1 parity gate (identical tokens). ⚠️ Known issue, still present on 26A428, affecting all Core AI LLMs (and how these bundles dodge it): the KV-write bug page.

CoreML (iOS 18+) variant of this model: qwen3.5-0.8B-CoreML.

<!-- funnel:v1 -->

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->