anziank/grio-qwen2.5-1.5b-coreml-anyLM-seq512

Qwen 2.5 1.5B Instruct — CoreML (seq=512 fixed, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from `Qwen/Qwen2.5-1.5B-Instruct` for use with HuggingFace's AnyLanguageModel Swift framework and `swift-transformers` ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

• —`grio-qwen2.5-1.5b-coreml-anyLM-seq512` (this repo) Fixed [1, 512] — short-form enhancement / single-sentence translation. Smallest graph. Runs on .all.
• —`grio-qwen2.5-1.5b-coreml-anyLM-seq1024` Fixed [1, 1024] — medium-context enhancement, multi-sentence translation. Runs on .all.
• —`grio-qwen2.5-1.5b-coreml-anyLM-seq2048` RangeDim [1, 1..2048] — variable-length / long-context. Must use `.cpuAndNE` or `.cpuAndGPU`; .all SIGSEGVs on the RangeDim variant.

Model details

Verified output (greedy, deterministic)

Smoke output from this .mlpackage. Useful as a regression marker if you re-convert or quantize.

• —Raw completion, cpu_and_gpu Prompt: "The capital of France is" Output: " Paris. The capital of France is also the capital of which country?"
• —ChatML System: "You are a helpful assistant. Answer concisely." User: "What is the capital of France?" Output: "Paris." (stops at <|im_end|> after 2 tokens)

This output is identical to the matching seq1024 and seq2048 variants in this collection, since the underlying weights and graph topology are the same; only the static seq dimension differs.

Observed performance (M1 Pro, macOS)

Benchmark notes: full-pad 512, 32-token greedy decode.

Smoke runs pad to the full 512-position graph every step, so the ms/tok number reflects fixed-shape compute over 512 positions — not the real-prompt-length cost you would see with the seq2048 RangeDim variant.

Runtime gotchas (please read before integrating)

1. 1.Fixed context window — input + output ≤ 512 tokens. Pad shorter prompts with zeros and set attentionMask = 0 on padded positions.
2. 2.`logits` is per-position, rank-3 [1, 512, 151936]. Pick the row at the last real prompt token for greedy/sampled decode.
3. 3.`.all` works on this fixed-shape variant — no SIGSEGV. The .all crash documented in the seq2048 variant is specific to RangeDim graphs.
4. 4.Qwen 2.5 uses two stop tokens in `generation_config.json`: <|endoftext|> (id 151643) and <|im_end|> (id 151645). tokenizer.eos_token_id returns only <|im_end|>. If your runtime stops only on the single tokenizer EOS id, it may miss the document-end token in some prompts. Read the full eos_token_id list from generation_config.json for robustness. swift-transformers ≥ 1.3 handles this correctly.

Conversion notes (for the CoreML community)

This model was produced from PyTorch source via torch.export + coremltools.convert. Findings worth flagging for anyone converting Qwen 2.5 (or similar HF causal LMs) to a swift-transformers-compatible CoreML format:

1. 1.Use `attn_implementation="sdpa"`. The HF default "eager" for Qwen 2.5 produces an op graph that triggers RuntimeWarning: overflow encountered in cast in coremltools/converters/mil/mil/passes/defs/optimize_repeat_ops.py:433 under RangeDim shapes, which silently corrupts logits. "sdpa" produces a cleaner op graph that lowers reliably. Fixed-shape (seq512 / seq1024) builds of Qwen 2.5 are not affected by the warning, but we still use SDPA here for consistency across the collection.
2. 2.Use `torch.export.default_decompositions()` — the all-decompositions mode ({}) can SIGSEGV at 1.5B+ scale under RangeDim.
3. 3.FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ on tokens where the model has near-tied top-1 candidates. Expected behavior, not a conversion bug.

Usage (Swift)

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .all,  // fixed shape — ANE-friendly
    chatTemplateHandler: { instructions, prompt in
        // Qwen 2.5 uses ChatML format; tokenizer.json's Jinja template applies the special tokens.
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a helpful assistant.")
let response = try await session.respond(to: "Improve this text: …")
print(response.content)

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 151936, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
).run_decompositions(torch.export.default_decompositions())

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, 512), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, 512), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Apache-2.0. Weights from `Qwen/Qwen2.5-1.5B-Instruct` by Qwen Team / Alibaba Cloud. Re-uploaded as a CoreML port; original model card terms apply.