sayedM/cohere-transcribe-arabic-cpu-friendly
cohere-transcribe-arabic — CPU-friendly (int8 + Arabic CTC draft head)
A CPU-only build of CohereLabs/cohere-transcribe-arabic-07-2026, a 2.07-billion-parameter Conformer encoder–decoder ASR model. No GPU, no CUDA.
Two things are packaged here that the original does not have:
- int8 dynamic quantization of all 515
nn.Linearmodules — 87.6% of the parameters — stored as safetensors rather than a pickle. - An Arabic CTC draft head (57.5M parameters) for speculative decoding, which makes short clips 1.42× faster at a cost of 0.04 percentage points of word error rate.
Together: 158.5 s → 4.82 s on a 33-second Arabic clip, on a six-core desktop CPU.
The draft head, and why it helps
The decoder here is autoregressive: one forward pass per token, ~130 per chunk. On a CPU at batch 1 that is not an arithmetic problem — it is a per-call problem. Measured on this machine:
Thirty-two tokens cost 1.30× what one token costs. The per-token price falls 24-fold. Almost all of a decoder call is fixed overhead, and the arithmetic in between is nearly free.
That is exactly the condition speculative decoding needs. If something cheap can guess the next K tokens, the decoder can check all of them in a single forward for roughly the price of checking one.
The CTC head is that cheap thing. It sits on the encoder output and is frame-synchronous and non-autoregressive — CTC assumes tokens are conditionally independent given the audio, so the whole hypothesis comes out of one forward pass. That independence assumption makes it a poor transcriber and an excellent drafter. Measured cost: 22.8 ms on a 10-second clip, against a 1069 ms decoder — a 2.1% tax.
The effect, measured: 2.66 tokens retired per decoder call instead of 1, which is 1.29×–1.48× wall clock depending on the clip, as shown in the figure above.
A wrong guess cannot produce a wrong word
The verifier keeps only the prefix its own argmax agrees with, then appends one token taken straight from the decoder. A drafted token reaches the output only if the decoder would have produced it anyway. A completely wrong draft degrades to ordinary greedy decoding. What a bad draft costs is time, not accuracy — which is why the accuracy column below barely moves.
Training data for the CTC head
Only the audio from that dataset was used. The targets are not its transcripts — they are what CohereLabs/cohere-transcribe-arabic-07-2026 itself emits on that audio. The drafter's job is to predict this model's output, so training it on anything else would teach it to propose tokens the verifier would reject. It also means no human annotation was involved anywhere in this head.
Labels were produced by an fp32 GPU replica of the deployed int8 model rather than by bf16, which cut label noise against the deployed model from 2.81% to 1.74% WER.
Vocabulary saturates almost immediately. Only 274 of the 16,384 vocabulary entries contain an Arabic character at all — the rest belong to other scripts the multilingual base model supports — and the tokenizer is sub-word for Arabic at ~2.4 tokens per word. After 0.83 hours, the types seen 10+ times already covered 99.24% of all token occurrences. Past that point more data buys acoustic quality, not new symbols.
The head is data-limited, not capacity-limited: acceptance went 0.143 at 0.83 h to 0.331 at 18.3 h, and test loss was still improving on the metric that matters when labelling stopped for time. More hours would help.
Where the time goes
Two independent measurements agree on this: forward hooks attributing time to architectural components, and torch.profiler attributing it to ATen kernels. The kernel view:
aminmax fires exactly once per linear_dynamic — that is dynamic quantization recomputing an activation scale on every single call. And 48,600 calls to pure-metadata operations do no arithmetic at all. The decoder issues roughly 3,520 of the 4,079 GEMM calls, each on a 1×1024 vector.
Performance
Measured on an Intel i5-12400F (6 cores / 12 threads, AVX2, no AVX-512, no AMX), 32 GB RAM, Windows 11, torch 2.14.0+cpu, transformers 5.16.1. Wall clock includes feature extraction, encoding, decoding and detokenization. RTFx = seconds of audio per second of compute; 1.0× is real time.
The three precisions, 33.3 s Arabic clip
bf16 is the checkpoint's native dtype and the worst possible choice on this class of CPU: AVX2 has neither AVX512-BF16 nor AMX, so PyTorch emulates it in software and the model runs five times slower than simply listening to the audio. That single line is the largest factor in the headline number.
End to end, 954 seconds of mixed Arabic and English
int8, batch 16, 12 threads, preallocated KV cache, draft head enabled:
What the draft head is worth, referenced to fp32
fp32 is the only reference that separates different from worse:
Accuracy cost of int8, by normalization level
Quote 2.06%, not 5.05%. Of 122 raw differences, 73 are orthographic convention — hamza seating (أ→ا), ta-marbuta (ة→ه), punctuation — not a different word being heard. Standard Arabic ASR evaluation normalizes these. Per file at the normalized level: two clips are identical to fp32, and the 14-minute recording — the only statistically meaningful one at 2,086 words — is 2.25%.
For reference: the same model on a GPU
A GPU is both faster and more accurate — bf16 is a far lighter perturbation than int8. This repo is for when there is no GPU. Worth knowing: the gap is 4.2× at batch 1 and 13.2× batched, because only the GPU gains from being fed more work. The draft head also transfers to the GPU, where it gives 1.40× at batch 1 — almost exactly what it gives on the CPU, for the same underlying reason.
Usage
pip install "transformers>=5.4" torch safetensors soundfile librosafrom huggingface_hub import snapshot_download
import sys
repo = snapshot_download("sayedM/cohere-transcribe-arabic-cpu-friendly")
sys.path.insert(0, repo) # the loader ships inside the repo
from cpu_model_loader import load_cpu_model, load_draft_head, transcribe
model, processor = load_cpu_model(repo)
head, _ = load_draft_head(repo) # optional; Arabic only
print(transcribe(model, processor, "audio.mp3", language="ar", head=head))Without the draft head, drop head=head. language is "ar" or "en" and is not optional in practice: an English recording decoded as Arabic turned the name Nasser into NASA.
Loading takes about 10 s and needs ~2.9 GB of RAM.
How the quantization works
Every nn.Linear — 1.81 B of the 2.07 B parameters — is stored as int8. Convolutions, layer norms and embeddings stay fp32. Weights use one per-tensor scale, float32(max|W| / 127.5), with a zero point of 0. Activations are quantized per call by quantized::linear_dynamic, to 7 bits (reduce_range), from the runtime minimum and maximum of whatever tensor the layer is handed.
That last detail has a visible consequence: there is one activation scale for the whole tensor, so handing a layer a wider tensor quantizes it more coarsely. This is why the model's output shifts slightly with batch size, and why speculative decoding cannot be bit-identical to greedy.
The draft head architecture: one ParakeetEncoderBlock copied from encoder.layers[47], then a projection, a LayerNorm and an output matrix copied from the base model's decoder.proj, decoder.norm and proj_out. Folding those three into a single matrix — the obvious "free" initialisation — measured worse than random init, because the real path contains a LayerNorm and a LayerNorm cannot be folded into a matrix product.
Limitations
- The draft head is Arabic-only. On English it drafts noise, every draft is rejected, and the wider verify forward makes it a net loss — measured 0.95×.
transcribe()applies it only to single-chunk audio; passhead=Nonefor other languages. - Speculative decoding is batch-1 only.
transformers' assisted generation does not support more. Long audio is already batched, which amortizes the decoder anyway. - Do not benchmark the draft head against int8 greedy. It differs by ~4% WER there, which is the quantizer's sequence-length sensitivity, not drafting error. Against fp32 the cost is 0.04 pp.
- CPU only.
quantized::linear_dynamicis registered for the CPU dispatch key alone; moving this model to CUDA raisesNotImplementedError. For a GPU, use the base model in bf16. - int8 costs accuracy. 2.06% WER from fp32. If you have the RAM and can spare the speed, fp32 is more faithful; if you have a GPU, bf16 is both faster and more accurate.
- The draft head saw 18.3 h of one dataset's dialect mix. Acoustics far from it will accept less.
- Inherits every limitation of the base model, including its language and domain coverage.
Verification
The export is checked against the original quantized model before publishing, by cpu_model_loader.verify():
- every one of the 515 quantized layers is present, with identical scales and
int_repr; - sampled layers dequantize to a maximum difference of 0.0;
- both models generate byte-identical token sequences on the same audio.
Two defects that check caught and that would otherwise have shipped silently: a non-persistent buffer (encode_positions.inv_freq) that state_dict() does not report, and a GenerationConfig rebuilt from config.json that lost decoder_start_token_id, which changed decoding while leaving the weights bit-perfect.
Files
License and attribution
Apache 2.0, inherited from CohereLabs/cohere-transcribe-arabic-07-2026, copyright Cohere Labs.
Changes made to the original (Apache 2.0 §4b): weights of all nn.Linear modules quantized to int8 as described above; a separately trained CTC draft head added; no change to the architecture, the tokenizer, or the training data of the base model.
The base repository is gated. This derivative is not, so if you need the original weights, please obtain them from Cohere Labs directly and accept their terms. Speech data for the draft head came from `oddadmix/lahgtna-v3-small`; please observe that dataset's own licence and terms for any redistribution of the audio itself (none of it is included here).