dawsonvosburg/cohere-transcribe-03-2026-coreml

Cohere Transcribe 03-2026 — Core ML (FP16 encoder, 256-token decoder cache)

A Core ML build of `CohereLabs/cohere-transcribe-03-2026` for Apple Silicon, derived from `FluidInference/cohere-transcribe-03-2026-coreml` with two changes: the encoder is re-exported at FP16 compute precision, and the decoder's KV cache is widened from 108 to 256 tokens.

Both changes exist to make a full 35-second window transcribable in one pass.

What's here

Compiled .mlmodelc rather than .mlpackage: this is what the runtime loads, and shipping it compiled skips a multi-second on-device compile at first load.

What changed, and why

1. Encoder re-exported at FP16 compute precision

The upstream Core ML encoder carries INT8 weights but requests FP32 compute precision. On Apple Silicon that forces a ~60 s MPSGraph JIT compile per process on the GPU (12.5 GB peak) and makes the CPU path dequantize to FP32. Re-exporting the same MIL program with compute_precision=FLOAT16 removes both: the GPU path runs at 0.08 s/window after a 10 s warm-up, and the CPU path runs in ~0.6 s per 35 s window at ~240 MB real footprint. The Neural Engine still rejects this conversion lineage; the intended production configuration is CPU compute units, which is faster end-to-end for a per-recording worker because it has no warm-up.

The weights are untouched — this is a round-trip through coremltools.converters.mil.frontend.milproto.load, which reconstructs the program through the MIL builder and re-runs type inference, then re-converts at FP16. No PyTorch and no original checkpoint are involved.

Script: `convert_encoder_fp16.py` — in the deriving repository at spikes/cohere-transcribe/convert_encoder_fp16.py.

2. Decoder KV cache widened 108 → 256

The upstream decoder's KV cache is dimensioned for 108 tokens, which caps generation well below what 35 seconds of speech needs; long windows come back truncated mid-sentence.

The cache dim stays static rather than becoming a RangeDim. This decoder is "cache-external" — the entire KV cache crosses the Core ML boundary on every decode step — so a symbolic sequence dim would force Core ML to re-specialize the graph on every token (a growing cache is a new shape each step) without saving any memory traffic. Making the static dim bigger is what actually removes the cap.

The method: the shipped MIL program is already shape-polymorphic in the cache dim (scatter_along_axis + matmul + softmax over the mask); the 108s in model.mil are only inferred type annotations. So the conversion patches the cache/mask sequence dim in the serialized proto — in both spec.description and spec.mlProgram.functions[].inputs — reloads the program through milproto.load so type inference re-runs from the new placeholder shapes, and re-converts at FP16 / iOS18.

Script: spikes/cohere-transcribe/convert_decoder_dyncache.py, run with --max-seq-len 256.

Runtimes read the cache width from the model, so the usable token budget follows automatically — 246 new tokens after the 10-token prompt.

Intended runtime

FluidAudio's CoherePipeline, via a fork carrying Cohere fixes not yet upstream (long-form seam handling, encoder warm-up at load, and sizing the decode loop from the decoder's actual KV cache — the last is what lets the 256-wide cache be used at all).

let models = try await CoherePipeline.loadModels(
    encoderDir: modelDir,
    decoderDir: modelDir,
    vocabDir: modelDir,
    decoderVariant: .v2,
    computeUnits: .all
)
let result = try await CoherePipeline().transcribe(
    audio: samples,          // 16 kHz mono Float32, ≤ 35 s
    models: models,
    language: .english
)

Audio must be 16 kHz mono Float32, at most 35 seconds — the encoder's single-chunk window. Longer audio needs sliding-window chunking with seam merging, which the runtime provides separately.

No timestamps. The model has no timestamp head; a transcription returns text and token ids only. This matches the GGUF build of the same model, whose transcribe_returned_timestamp_kind reports none. Word-level timing has to come from elsewhere.

License and provenance

Apache-2.0, inherited from the base model.

• —Base model: `CohereLabs/cohere-transcribe-03-2026` (Apache-2.0)
• —Core ML conversion this derives from: `FluidInference/cohere-transcribe-03-2026-coreml`
• —Changes here: FP16 compute-precision re-export of the encoder; decoder KV cache widened to 256 via a MIL proto patch. Weights are unmodified in both cases.

Refer to Cohere's model card for training data, intended use, evaluation methodology, and limitations. Nothing here re-evaluates the model's accuracy; these are packaging changes.

Uploading

hf upload dawsonvosburg/cohere-transcribe-03-2026-coreml . --repo-type model

Run from this directory. .mlmodelc bundles are directories, so this uploads them file by file; the copies here are real files rather than symlinks because hf upload does not follow symlinks reliably.