Modelpublic

smdesai/canary-1b-v2-pal6-int8-coreml

sourceHugging Facecc-by-4.0updated 18d agoView on Hugging Face

0likes

Model Card

Canary 1B v2 — CoreML (6-bit encoder + INT8 decoder, KV-cache)

Smallest Apple Neural Engine-resident CoreML build of NVIDIA canary-1b-v2: a 6-bit k-means palettized encoder paired with an INT8 per-channel stateful KV-cache decoder and cross-KV. Accuracy matches the FP16 reference build (canary-1b-v2-coreml) at under half its RAM. Pick this over the all-INT8 build (canary-1b-v2-int8full-coreml) when the extra ~7 ms/window encoder cost and the longer first-load compile are acceptable.

Base model: nvidia/canary-1b-v2 (NVIDIA NeMo EncDecMultiTaskModel, 1B parameters, 25 European languages, ASR + speech translation). License: the base model is released under CC-BY-4.0; this conversion carries the same license. Please attribute NVIDIA for the model.

This build


Encoder	6-bit k-means palette, 16-channel groups; Neural Engine-resident, 614 MB on disk, ~0.63 GB resident
Decoder / cross-KV	INT8 per-channel, 136 MB + 17 MB; ~134 MB Neural Engine-resident
Download	0.77 GB
iOS RAM while transcribing	910 MB (measured) vs 1.9 GB for the FP16 build
Encoder latency	~43 ms per 15 s window (M3 Max, warm); decoder step 2.4–3.3 ms per token

Placement note: load with MLComputeUnits.cpuAndNeuralEngine; under .all the planner prefers the GPU for the INT8 decoder.

Build family

All 1B v2 builds share the same preprocessor, tokenizer, package layout, and decode contract; only the weight format of the encoder and/or decoder differs. Download sizes are as hosted on the Hub; iOS RAM was measured in an app while transcribing a 6-minute file.

Repo	Build	Encoder weights	Decoder weights	Download	iOS RAM
`canary-1b-v2-coreml`	FP16 (reference)	FP16	FP16	1.89 GB	1.9 GB (measured)
`canary-1b-v2-int8-coreml`	INT8 encoder	INT8 per-channel	FP16	1.10 GB	1.2 GB (measured)
`canary-1b-v2-pal6-coreml`	6-bit encoder	6-bit palette, g=16	FP16	0.92 GB	1.0 GB (measured)
`canary-1b-v2-int8full-coreml`	INT8 full	INT8 per-channel	INT8 per-channel	0.95 GB	1.0 GB (measured)
`canary-1b-v2-pal6-int8-coreml` (this repo)	6-bit + INT8	6-bit palette, g=16	INT8 per-channel	0.77 GB	910 MB (measured)
`canary-180m-flash-coreml`	180M Flash	FP16 (17L, d=512)	FP16 (4L)	0.37 GB	470 MB (measured)

Files

File	Contents
`canary_preprocessor.mlmodelc`	FP32 mel front end (128 mel bins, 16 kHz mono, 15 s window)
`canary_encoder.mlmodelc`	FastConformer encoder, 32 layers, d=1024, subsampling 8
`canary_cross_kv.mlmodelc`	Cross-attention K/V projection, 8 layers (run once per window)
`canary_decoder_kv.mlmodelc`	Stateful single-step Transformer decoder, 8 layers, 16384-way LM head
`canary_spe.model`	SentencePiece tokenizer (16384 pieces)
`metadata.json`	Shapes, decode geometry, seed tokens, and the weight recipe of this build

Credits

Model: NVIDIA NeMo team, canary-1b-v2, CC-BY-4.0.