Modelpublic

smdesai/canary-1b-v2-int8-coreml

sourceHugging Facecc-by-4.0updated 17d agoView on Hugging Face

0likes8downloads

Model Card

Canary 1B v2 — CoreML (INT8 encoder, KV-cache)

Memory-optimized CoreML build of NVIDIA canary-1b-v2 for Apple silicon: the encoder's weights are quantized to INT8 per-channel (linear symmetric) and decompressed in flight on the Apple Neural Engine, so the encoder's resident memory halves with the same FP16 activations and Neural Engine placement as the FP16 reference build (canary-1b-v2-coreml). Decoder and cross-KV stay FP16. Accuracy is within noise of FP16.

Base model: nvidia/canary-1b-v2 (NVIDIA NeMo EncDecMultiTaskModel, 1B parameters, 25 European languages, ASR + speech translation). License: the base model is released under CC-BY-4.0; this conversion carries the same license. Please attribute NVIDIA for the model.

This build


Encoder	INT8 per-channel weights, Neural Engine-resident, 792 MB on disk, ~0.8 GB resident
Decoder / cross-KV	FP16 (unchanged), 271 MB + 34 MB
Download	1.10 GB
iOS RAM while transcribing	1.2 GB (measured) vs 1.9 GB for the FP16 build
Encoder latency	~36 ms per 15 s window (M3 Max, warm)

Build family

All 1B v2 builds share the same preprocessor, tokenizer, package layout, and decode contract; only the weight format of the encoder and/or decoder differs. Download sizes are as hosted on the Hub; iOS RAM was measured in an app while transcribing a 6-minute file.

Repo	Build	Encoder weights	Decoder weights	Download	iOS RAM
`canary-1b-v2-coreml`	FP16 (reference)	FP16	FP16	1.89 GB	1.9 GB (measured)
`canary-1b-v2-int8-coreml` (this repo)	INT8 encoder	INT8 per-channel	FP16	1.10 GB	1.2 GB (measured)
`canary-1b-v2-pal6-coreml`	6-bit encoder	6-bit palette, g=16	FP16	0.92 GB	1.0 GB (measured)
`canary-1b-v2-int8full-coreml`	INT8 full	INT8 per-channel	INT8 per-channel	0.95 GB	1.0 GB (measured)
`canary-1b-v2-pal6-int8-coreml`	6-bit + INT8	6-bit palette, g=16	INT8 per-channel	0.77 GB	910 MB (measured)
`canary-180m-flash-coreml`	180M Flash	FP16 (17L, d=512)	FP16 (4L)	0.37 GB	470 MB (measured)

Files

File	Contents
`canary_preprocessor.mlmodelc`	FP32 mel front end (128 mel bins, 16 kHz mono, 15 s window)
`canary_encoder.mlmodelc`	FastConformer encoder, 32 layers, d=1024, subsampling 8
`canary_cross_kv.mlmodelc`	Cross-attention K/V projection, 8 layers (run once per window)
`canary_decoder_kv.mlmodelc`	Stateful single-step Transformer decoder, 8 layers, 16384-way LM head
`canary_spe.model`	SentencePiece tokenizer (16384 pieces)
`metadata.json`	Shapes, decode geometry, seed tokens, and the weight recipe of this build

Credits

Model: NVIDIA NeMo team, canary-1b-v2, CC-BY-4.0.