anziank/grio-llama-3.2-1b-coreml-anyLM-seq1024

Llama 3.2 1B Instruct — CoreML (seq=1024 fixed, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from `meta-llama/Llama-3.2-1B-Instruct` for use with HuggingFace's AnyLanguageModel Swift framework and `swift-transformers` ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

• —`grio-llama-3.2-1b-coreml-anyLM-seq512` Fixed [1, 512] — short-context transcript cleanup and post-processing. Fastest Llama profile.
• —`grio-llama-3.2-1b-coreml-anyLM-seq1024` (this repo) Fixed [1, 1024] — medium-context copy-editing, translation. Always pays for full 1024 positions per step.

Model details

Verified output (greedy, deterministic)

Smoke output from this .mlpackage under .cpuAndGPU. Output is byte-identical to the matching seq512 variant in this collection (same weights, same graph topology; only the static seq dimension differs).

• —ChatML System: "You are a helpful assistant. Answer concisely." User: "What is the capital of France?" Output: "The capital of France is Paris." (stops at <|eot_id|> after 8 tokens)

Observed performance (M1 Pro, macOS)

Benchmark notes: full-pad 1024, 32-token greedy decode.

Runtime gotchas (please read before integrating)

1. 1.Fixed context window — input + output ≤ 1024 tokens. Pad shorter prompts with zeros and set attentionMask = 0 on padded positions.
2. 2.`logits` is per-position, rank-3 [1, 1024, 128256]. Pick the row at the last real prompt token for greedy/sampled decode.
3. 3.Llama 3.2 uses two stop tokens in `generation_config.json`: <|end_of_text|> (id 128001) and <|eot_id|> (id 128009). tokenizer.eos_token_id returns <|eot_id|>. If your runtime stops only on the single tokenizer EOS id, it may decode past <|eot_id|> into out-of-distribution territory. Make sure your generation loop checks both.
4. 4.Pay for full pad dimension: Shorter prompts are padded to the full sequence length, so you pay for the full static shape cost per step. If your typical prompts stay below 512 tokens, prefer the seq512 variant in this collection.

Conversion notes (for the CoreML community)

Findings worth flagging for anyone converting Llama 3.2 to CoreML:

1. 1.Use `attn_implementation="sdpa"`. SDPA produces a much cleaner MIL graph that lowers reliably to both GPU and Apple Neural Engine.
2. 2.Use `torch.export.default_decompositions()` to ensure standard graph representation and avoid conversion scaling errors during export.
3. 3.FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ slightly on tokens where top-1 candidates are near-tied. This is expected behavior.

Usage (Swift)

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .all,
    chatTemplateHandler: { instructions, prompt in
        // Llama 3.2 Instruct uses the tokenizer-owned header/chat template;
        // tokenizer_config.json and tokenizer files must stay alongside the model
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a helpful assistant.")
let response = try await session.respond(to: "Improve this text: ...")
print(response.content)

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 128256, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
).run_decompositions(torch.export.default_decompositions())

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, 1024), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, 1024), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Llama 3.2 Community License. Weights from `meta-llama/Llama-3.2-1B-Instruct` by Meta.