experimentalmachines/Qwen2.5-1.5B-Instruct-ExecuTorch

Qwen2.5-1.5B-Instruct for ExecuTorch

ExecuTorch exports of Qwen/Qwen2.5-1.5B-Instruct (revision 989aa7980e4c) for on-device inference with the openweights Android app or any ExecuTorch 1.4.0 runtime.

Files

Every backend is exported at every context window the runner could build (2k to 32k). The window is fixed inside the file: the runtime allocates the whole KV cache at load, so pick the largest window the device can hold (fits_phone_budget in each folder's config.json is the estimate against a 5 GB budget).

Tokenizer: `tokenizer.json`, copied unchanged from the source repo. Each backend folder has a config.json listing every window as a variant with the metadata the .pte reports, and an export-report-<window>.json per file with the full export record.

Memory

• —XNNPACK (CPU) at 2,048 tokens: the KV cache costs 57,344 bytes per token (fp32), 117,440,512 bytes for the whole window, allocated in full when the model loads.
• —XNNPACK (CPU) at 4,096 tokens: the KV cache costs 57,344 bytes per token (fp32), 234,881,024 bytes for the whole window, allocated in full when the model loads.
• —XNNPACK (CPU) at 8,192 tokens: the KV cache costs 57,344 bytes per token (fp32), 469,762,048 bytes for the whole window, allocated in full when the model loads.
• —XNNPACK (CPU) at 16,384 tokens: the KV cache costs 57,344 bytes per token (fp32), 939,524,096 bytes for the whole window, allocated in full when the model loads.

How it was made

• —XNNPACK (CPU) any arm64: ExecuTorch 1.4.0 export_llm: 8-bit dynamic activations and 4-bit weights in groups of 32, int8 per-channel embeddings, XNNPACK with extended ops, prefill chunk 2048, fp32 KV cache. Built by run 1.

License

A quantized derivative of Qwen/Qwen2.5-1.5B-Instruct, distributed under the same terms (apache-2.0). The upstream license files are included unchanged: `LICENSE`.