Modelpublic

mlboydaisuke/Streaming-Sortformer-Diar-CoreAI

sourceHugging Facecc-by-4.0updated 7d agoView on Hugging Face

2likes480downloads

Model Card

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark, the on-device LLM leaderboard.

Streaming Sortformer 4-spk v2 — Core AI

`nvidia/diar_streaming_sortformer_4spk-v2` (cc-by-4.0, 117M) converted to Apple Core AI — streaming speaker diarization ("who spoke when", up to 4 speakers) running fully on-device via the zoo. Only the neural core is a graph; the NeMo 128-mel frontend, the streaming chunk loop, and the AOSC speaker-cache compression run in the Swift host — a 1:1 port of NeMo sortformer_modules.py (inference path).

⚠️ Use the streaming v2 checkpoint (cc-by-4.0). The offline diar_sortformer_4spk-v1 is CC-BY-NC.

Files

—sortformer_float16.aimodel — the static forward_for_export core, fp16 (~237 MB, macOS GPU).
—sortformer_float16.h18p.aimodelc — the same graph AOT-compiled for iPhone (h18p, ~450 MB).
—sortformer_mel_filters_128x257.f32 — librosa-slaney mel filterbank (host log-mel frontend).
—metadata.json — streaming params + the fixed-buffer graph contract.

Fixed-buffer graph contract

inputs:  chunk_mel [1,1520,128]   host zero-pads each mel chunk
         spkcache  [1,188,512]     host-maintained speaker cache
         valid     [1,378]         1 = real frame / 0 = pad (spkcache block [0:len], chunk block [188:188+pe_len])
outputs: preds     [1,378,4]       sigmoid speaker activity
         chunk_pe  [1,190,512]     pre-encode embeddings (host appends them to the speaker cache)

Host: NeMo 128-mel (preemph 0.97 → STFT nfft=512/win=400/hop=160 → slaney mel → log, normalize=NA) → chunk the mel (188·8 frames, ±1 subsample ctx) → run the graph → slice chunk preds → `streamingupdate + compress_spkcache` (AOSC) → threshold 0.5/frame/speaker (frame = 80 ms) → turns.

Verification

Byte-gated vs NeMo forward_streaming at 100.00 % speaker-activity agreement (@0.5) on a 21.5 s and a 64.5 s clip (the latter exercises the AOSC cache compression ~4×), in Python, in Swift on Mac GPU, and on iPhone 17 Pro (A19 Pro, AOT h18p) — all driving this exported fp16 graph.

Use

Ships in the coreai-audio app (Transcribe tab, "Diarize — who said what"): the diarizer segments each speaker turn, then the on-device ASR (Whisper / Qwen3-ASR / Parakeet / Nemotron) transcribes it into a diarized transcript — Speaker 1 [0.3–4.1s]: …. Speaker diarization already ships on-device elsewhere (e.g. CoreML/ANE); this is speed parity, offered as a diarized transcript wired to the zoo's own ASR. Conversion + Swift host loop: see `conversion/sortformer_diar`.

Derived from NVIDIA's diar_streaming_sortformer_4spk-v2 (CC-BY-4.0); this Core AI conversion is released under the same license.

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.