mlboydaisuke/VibeVoice-Realtime-0.5B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) --> This model has no row on DeviceMark, the on-device LLM leaderboard. <!-- gen-cards:devicemark end -->

VibeVoice-Realtime-0.5B — Core AI

`microsoft/VibeVoice-Realtime-0.5B` (MIT) converted to Apple Core AI — the zoo's first multi-speaker / dialogue (podcast-style) TTS. iPhone (AOT) + Mac, all-fp16.

Not a "first on-device VibeVoice" claim — other CoreML/GGUF ports exist. What this is: the zoo's first multi-speaker TTS, app-integrated, and the other half of a generate -> diarize loop with the zoo's Streaming Sortformer diarizer.

Architecture

Dual Qwen2.5 LM (4-layer text context LM, norm = Identity + 20-layer speech trunk) -> per-frame next-token diffusion (4-layer adaLN head, DDPM cosine, v-prediction, DPMSolver++ 5-step, CFG 1.5) -> causal-conv acoustic VAE decoder (7.5 Hz latent -> 24 kHz, 3200 samples/frame). The LM predicts one latent per frame; the diffusion head denoises it; the VAE decoder renders audio. Multi-speaker output is host turn-switching: each Speaker N: turn is generated from its own voice preset and the turns are concatenated — no multi-speaker prefill, no acoustic encoder.

Contents

fp16 is required. int8 LMs diverge inside the speech feedback loop (min cos 0.187, early EOS); the diffusion head is fp16-sensitive too (pure-torch fp16 collapses to 0.79 — Core AI keeps the RMSNorm/adaLN reductions in fp32, so the host DDPM reference must run fp32).

Fixed shapes only. Every graph is static (q=1 decode, fixed-T decoder), so the runtime must not be given the expectFrequentReshapes hint on iOS: it makes the runtime skip the AOT specialization and compile on device, which segfaults inside the MPSGraph AICode compiler.

Gates

On device: 6 graph loads in 2.6 s (warm), 24 latents / 3.20 s of audio in 2.3 s = 10.6 tok/s ~ 1.4x real-time.

Use it

import CoreAIKit

let dialogue = try await KitDialogue(catalog: "vibevoice-realtime-0.5b")
let (audio, turns) = try await dialogue.perform("""
    Speaker 1: Did you know this runs entirely on the phone?
    Speaker 2: No cloud at all? That is wild.
    """)

Swift host reference: ondevice/VibeVoiceRunner (Mac) and VibeVoiceSelfTest.swift in the zoo's coreai-audio app — raw Core AI stateful-KV loop + a Swift DPMSolver++ sampler. Python host + conversion recipe: `conversion/vibevoice` (host_e2e.py = the full generate loop, host_multispeaker.py = the dialogue demo).

Base model: microsoft/VibeVoice-Realtime-0.5B (MIT). EN/ZH. Community port — not an Apple model.

<!-- funnel:v1 -->

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->