anziank/grio-qwen2.5-1.5b-coreml-anyLM-seq2048

Qwen 2.5 1.5B Instruct — CoreML (seq≤2048, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from `Qwen/Qwen2.5-1.5B-Instruct` for use with HuggingFace's AnyLanguageModel Swift framework and `swift-transformers` ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

• —`grio-qwen2.5-1.5b-coreml-anyLM-seq512` Fixed [1, 512] — short-form enhancement. Smallest graph. Runs on .all.
• —`grio-qwen2.5-1.5b-coreml-anyLM-seq1024` Fixed [1, 1024] — medium-context enhancement, multi-sentence translation. Runs on .all.
• —`grio-qwen2.5-1.5b-coreml-anyLM-seq2048` (this repo) RangeDim [1, 1..2048] — variable-length translation / long-context; fastest on short prompts since attention scales with actual length. Must use `.cpuAndNE` or `.cpuAndGPU`; .all SIGSEGVs (see Runtime gotchas).

Model details

Verified output (greedy, deterministic)

Smoke outputs from this .mlpackage. Useful as a regression marker if you re-convert or quantize.

• —Raw completion, cpu_and_gpu Prompt: "The capital of France is" Output: " Paris. The capital of France is also the capital of which country?"
• —ChatML System: "You are a helpful assistant. Answer concisely." User: "What is 2+2?" Output: "2+2 equals 4." (EOS at 7 tokens)
• —ChatML translation System: "You are a French translator. Translate the user message to French. Output only the translation." User: "The capital of France is Paris." Output: "La capitale de la France est Paris." (EOS at 9 tokens)
• —ChatML translation System: "You are a French translator. Translate the user message to French. Output only the translation." User: "The quick brown fox jumps over the lazy dog." Output: "Le renard brun rapide saute par-dessus le chien paresseux." (byte-identical to PyTorch FP16 reference)

Observed performance (M1 Pro, macOS)

Benchmark notes: real prompt about 27 tokens.

iPad device perf not benchmarked in this build run.

Runtime gotchas (please read before integrating)

1. 1.`.all` compute units crash this RangeDim model at inference. This is structural to RangeDim + ANE-mixed compute inside Core ML's MPSGraph / E5 program library on iOS 18 / macOS 15. The fixed-shape seq512 / seq1024 variants in this collection do NOT have this issue. If your runtime currently defaults to .all, route this RangeDim variant to .cpuAndGPU or .cpuAndNE explicitly.
2. 2.`logits` shape metadata is empty in the .mlpackage description (a coremltools artifact for RangeDim outputs). Verify at runtime with a real predict() call — the actual output is rank-3 [1, seq_len, 151936] and works with swift-transformers' assert(scores.rank == 3).
3. 3.Qwen 2.5 uses two stop tokens in `generation_config.json`: <|endoftext|> (id 151643) and <|im_end|> (id 151645). tokenizer.eos_token_id returns only <|im_end|>. If your runtime stops only on the single tokenizer EOS id, it may miss the document-end token in some prompts. Read the full eos_token_id list from generation_config.json (bundled in this repo) for robustness. swift-transformers ≥ 1.3 handles this correctly.

Conversion notes (for the CoreML community)

This model was produced from PyTorch source via torch.export + coremltools.convert. Findings worth flagging for anyone converting Qwen 2.5 (or similar HF causal LMs) to a swift-transformers-compatible CoreML format:

1. 1.`attn_implementation` silently corrupts RangeDim graphs. With attn_implementation="eager" (the HF default for Qwen 2.5), coremltools raises RuntimeWarning: overflow encountered in cast in coremltools/converters/mil/mil/passes/defs/optimize_repeat_ops.py:433 at pass ~65 of the MIL default pipeline when the input shape contains RangeDim. Conversion completes silently. ABI verifies as rank-3 [1, S, 151936]. But model.predict() returns degenerate logits — greedy argmax collapses to a single low-index token on every step. Switching to attn_implementation="sdpa" produces a different op graph (3667 MIL ops vs eager's 3443) that skips the buggy pass and produces correct outputs. Fixed-shape (`seq512` / `seq1024`) builds are not affected by the warning, presumably because the overflow is on RangeDim bounds arithmetic. We still use sdpa for the fixed-shape builds for consistency.
2. 2.`run_decompositions({})` SIGSEGVs at 1.5B with RangeDim. Use torch.export.default_decompositions() instead. Same optimize_repeat_ops neighborhood, but as a hard crash rather than a silent corruption.
3. 3.`.mlpackage` output description for `RangeDim` models reports empty `logits` shape metadata. Verify rank with a real predict() probe at conversion time — do not trust descriptor introspection alone.
4. 4.FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ on tokens where the model has near-tied top-1 candidates (e.g., picking "Kaffeetasse" vs "Kaffeekanne" for "a coffee" in German translation). Expected behavior, not a conversion bug.

Usage (Swift)

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .cpuAndNE,  // or .cpuAndGPU — NOT .all on this RangeDim variant
    chatTemplateHandler: { instructions, prompt in
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a French translator. Output only the translation.")
let response = try await session.respond(to: "The capital of France is Paris.")
print(response.content) // "La capitale de la France est Paris."

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",         # CRITICAL for RangeDim — see Conversion notes
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
seq = torch.export.Dim("sequence_length", min=1, max=2048)
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 151936, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
    dynamic_shapes={"inputIds": {1: seq}, "attentionMask": {1: seq}},
).run_decompositions(torch.export.default_decompositions())  # NOT {}

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, ct.RangeDim(1, 2048)), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, ct.RangeDim(1, 2048)), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Apache-2.0. Weights from `Qwen/Qwen2.5-1.5B-Instruct` by Qwen Team / Alibaba Cloud. Re-uploaded as a CoreML port; original model card terms apply.