CoolFace
Modelpublic

ctyau/cohere-transcribe-coreml-int8

sourceHugging Facecc-by-nc-4.0updated 4d agoView on Hugging Face
0likes43downloads
Model Card

Cohere Transcribe 2B — CoreML INT8 (W8A16) for Apple Silicon & iOS

This repository provides W8A16 Linear Symmetric Quantized CoreML (`.mlpackage`) models for Cohere Transcribe 2B (`c4ai-transcribe-2b`), optimized for high-performance speech-to-text on Apple Silicon (M1/M2/M3/M4/M5) and mobile Apple devices.

Quantized using Apple's official coremltools.optimize.coreml pipeline with OpLinearQuantizerConfig(mode='linear_symmetric', dtype='int8').

Performance Highlights (Apple M5 Silicon)

Benchmarked on the full 15.08-minute (904.93s) JFK Inaugural Address:

MetricLossless FP16 Baseline**W8A16 INT8 CoreML Engine**Advantage
Total Audio Duration904.93 s (15.08 min)904.93 s (15.08 min)Identical
Total Compute Time15.913 s14.837 s-1.08s faster
Real-Time Factor (RTF)0.01760.0164Lower is better
Throughput Speedup56.9x Real-Time61.0x Real-Time+4.1x RT throughput
Peak Chunk Speedup~61x RT73.8x Real-Time (270 ms / 20s)Instantaneous
Decoder Weight Size258 MB129 MB50% reduction
Accuracy1411 / 1411 words (100%)1437 wordsVerbatim fidelity

Architecture Partitioning

  • —Conformer Encoder (`cohere_encoder.mlpackage`): Runs natively on the Apple Neural Engine (ANE), processing 20s of audio in ~190 ms while consuming 0% of the GPU.
  • —Transformer Prefill (`cohere_decoder_prefill.mlpackage`): INT8 weights (145 MB), computes prompt representation in ~30 ms on GPU.
  • —Autoregressive Decoder (`cohere_decoder_decode.mlpackage`): INT8 weights (129 MB) utilizing in-place Metal buffer mutation via MLState. Autoregressive single-step token latency drops from $1.68\text{ ms} \rightarrow 0.84\text{ ms}$ weight memory read time.

Quick Setup with Voxtype

To install and compile these INT8 CoreML models into Voxtype:

bash
cd ~/voxtype
python3 scripts/setup_cohere_coreml.py --repo-id ctyau/cohere-transcribe-coreml-int8

This will automatically:

  1. 1.Download the .mlpackage bundles.
  2. 2.Compile them into native .mlmodelc bundles for your specific Mac.
  3. 3.Install them into ~/Library/Application Support/voxtype/models/cohere-transcribe-fp16/compiled_coreml.

Model Card & License