Modelpublic

mlboydaisuke/BitVLA-CoreAI

sourceHugging Facemitupdated 8d agoView on Hugging Face

0likes60downloads

Model Card

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark, the on-device LLM leaderboard.

BitVLA (1.58-bit Vision-Language-Action) — Core AI

The first Vision-Language-Action model running fully on-device on iPhone, via Apple Core AI. A Core AI conversion of `lxsy/bitvla-bf16` (BitVLA, arXiv:2506.07530, MIT).

BitVLA takes an image + a natural-language instruction and predicts a 7-DoF robot end-effector action (Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper) — OpenVLA-style discrete action tokens. Every transformer weight, in both the BitNet b1.58-2B language model and the BitSigLIP-SO400M vision tower, is 1.58-bit ternary ({-1, 0, +1}) — ~32× smaller than a full-precision VLA (OpenVLA-7.5B ≈ 15 GB), so the whole policy fits and runs on a phone GPU. The language model's per-layer linears run a custom 2-bit packed-ternary Metal kernel.

Part of the Core AI model zoo — on-device AI for iPhone & Mac through Apple Core AI: https://github.com/john-rocky/coreai-model-zoo

On-device (iPhone 17 Pro — Core AI GPU, greedy)

One image + instruction → 7-DoF action:

stage	warm
vision encode (BitSigLIP-SO400M, 256 tokens)	0.13 s
LLM prefill (≈308 positions, M=1 ternary kernel)	8.8 s
action decode (7 tokens)	0.26 s

Resident ≈ 2 GB, no jetsam. On-device output matches the official model: 6/7 action tokens identical, 7-DoF action effectively identical, vision embeddings at per-token cosine 0.999.

What's in this repo

—`h18p/` — device-ready, AOT-compiled for the iPhone 17 Pro (h18p) GPU: bitvla_vision/ (BitSigLIP tower), bitvla_llm_act/ (BitNet LLM, 256-row action head + ternary kernel), bitvla_device_data/ (preset-instruction text embeds + the 256-row action-token embed table + norm_stats + a sample image — so the device needs no tokenizer or embedding table).
—`aimodel/` — portable source .aimodels (vision + LLM); re-AOT for another device with xcrun coreai-build compile <…>.aimodel --platform iOS --preferred-compute gpu --architecture <arch>.

Architecture

—LLM = BitNet b1.58 2B4T (30L, hidden 2560, FFN 6912, GQA 20/5 hd128, ReLU² FFN, SubLN, RoPE θ500000), W1.58-A8 (per-tensor absmean ternary weight + per-token int8 activation).
—Vision = BitSigLIP-SO400M (26L, hidden 1152, FFN 4304, patch14/224 → 256 tokens), ternary linears; fp16 activations on device.
—Connector = 2-layer MLP (1152→2560→2560), fp16.
—Action = OpenVLA discrete: 256 vision embeds spliced into the LLM; 7 action tokens from the vocab tail → 256-bin → BOUNDS-Q99 un-normalization (norm_stats, 27-dataset OXE mix; e.g. unnorm_key = bridge_orig).

Conversion & how it works

Recipe, kernel notes, and the device gotchas (custom kernel must be AOT-compiled — it can't JIT on device; the dynamic-shape LLM .aimodelc loads with expectFrequentReshapes=false; vision uses fp16 activations because the in-graph A8 quant stalls the GPU) are in the zoo:

—card: https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/bitvla.md
—knowledge: https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/bitvla-1.58bit-vla.md
—conversion: https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion

License

MIT, inheriting `lxsy/bitvla-bf16` / BitVLA. This is a converted redistribution of the Core AI artifacts; see the base model for original terms.

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.