RockTalk/Lance-3B-Video-MLX

Lance-3B-Video-MLX

Video variant of Lance-3B-MLX. First native MLX port of ByteDance Research's Lance — a 3 B-parameter unified multimodal model for image/video generation, editing, and understanding. Runs natively on Apple Silicon, no CUDA required.

The architecture is Qwen2.5-VL-3B + parallel MoE-gen experts + Wan 2.2 VAE. Lance uses a "Mixture-of-Tokens" routing: every attention block and MLP has a parallel *_moe_gen branch. Text tokens go through normal weights; VAE-latent (generation) tokens go through the _moe_gen weights, in the same forward pass.

Quick start (self-contained — no external repo needed)

# 1. Download the model (one-time, ~30 GB total)
hf download RockTalk/Lance-3B-Video-MLX --local-dir Lance-3B-Video-MLX

# 2. Install runtime deps
cd Lance-3B-Video-MLX
pip install -r requirements.txt

# 3. Generate a 9-frame video (T_lat=3 → 9 output frames)
python inference.py --prompt "a calm ocean wave rolling onto a sandy beach"

# Longer video (29 frames):
python inference.py --prompt "..." --t-lat 8

# Pure image (T_lat=1):
python inference.py --prompt "..." --t-lat 1 --size 512 --steps 30

First run auto-fetches the companion VAE (`RockTalk/Wan2.2-VAE-MLX`, ~2.6 GB, cached as wan22_vae.safetensors) so all subsequent runs are fully offline.

CLI options

python inference.py \
  --prompt "..." \
  --out output.png       # frame strip + per-frame PNGs saved next to it
  --size 256             # 256 recommended for T2V
  --t-lat 3              # latent frames; output = (t_lat-1)*4 + 1 frames
  --steps 24             # 24 typical for T2V
  --cfg 4.0
  --seed 0
  --mp4                  # also emit output.mp4 (needs `pip install 'imageio[ffmpeg]'`)
  --fps 8

Frame-count table: T_lat=1 → 1, T_lat=3 → 9, T_lat=8 → 29, T_lat=31 → 121 (max).

Programmatic use

from inference import build_lance_config, ensure_vae_weights
from lance_mlx.lance import Lance, LanceConfig
from lance_mlx.vae_wan22 import Wan2_2_VAE
from transformers import AutoTokenizer
import json, mlx.core as mx
from pathlib import Path

repo = Path(".")
cfg_json = json.loads((repo / "config.json").read_text())
lance_cfg = build_lance_config(cfg_json)

model = Lance(lance_cfg)
weights = mx.load("model.safetensors")
non_vit = {k: v for k, v in weights.items() if not k.startswith("vit_model.")}
model.load_weights(list(non_vit.items()), strict=True)

vae = Wan2_2_VAE(z_dim=48, c_dim=160, dim_mult=(1, 2, 4, 4),
                 temperal_downsample=(False, True, True))
vae.model.load_weights(list(mx.load(str(ensure_vae_weights(repo))).items()), strict=True)

tok = AutoTokenizer.from_pretrained(".")
text_ids = mx.array(tok("a calm ocean wave", add_special_tokens=False,
                        return_tensors="np").input_ids[0], dtype=mx.int32)

latent = model.sample_t2i(
    prompt_token_ids=text_ids,
    latent_shape=(3, 16, 16),    # T_lat=3 → 9 output frames @ 256×256
    special_token_ids={"bos": 151644, "eos": 151645,
                       "start_of_image": 151652, "end_of_image": 151653,
                       "image_token_id": 151655},
    num_steps=24, timestep_shift=3.5, cfg_scale=4.0, seed=0,
)
video = vae.decode(latent)        # (1, 9, 256, 256, 3) in [-1, 1]

What works

Sample generations

Text-to-video (T2V)

Verified on M4 Studio (128 GB). 24 steps, CFG=4, 256×256, T_lat=3 → 9 frames:

"a calm ocean wave rolling onto a sandy beach" — 9-frame strip (left-to-right):

[image]

Single frames (frame 0, 4, 8):

Performance

Measured on M4 Studio (128 GB) at CFG=4 (one conditional + one unconditional forward per step):

First-call kernel-compile penalty: ~few seconds per new resolution.

Differences vs Lance-3B-MLX

This is the same architecture as the image variant, with two differences:

• —model.safetensors: 26.5 GB (vs 23 GB) — extra weights for multi-frame attention
• —latent_pos_embed.pos_embed: 31 × 64 × 64 = 126,976 positions (vs 1 × 64 × 64 = 4,096) — supports up to 31 latent frames (≈ 121 video frames @ 4× temporal downsample)

T2I via this checkpoint works the same as Lance-3B-MLX. T2V is now live — uses the Wan 2.2 VAE v0.1.0 streaming cache under the hood. Pass latent_shape=(T_lat, H_lat, W_lat) with T_lat > 1 to sample_t2i to generate a video.

Files

How the MoE-gen routing is implemented in MLX

Lance's checkpoint contains two sets of weights per Qwen2 block:

self_attn.{q,k,v,o}_proj         self_attn.{q,k,v,o}_proj_moe_gen
self_attn.{q,k}_norm             self_attn.{q,k}_norm_moe_gen
mlp.{gate,down,up}_proj          mlp_moe_gen.{gate,down,up}_proj
input_layernorm                  input_layernorm_moe_gen
post_attention_layernorm         post_attention_layernorm_moe_gen

For T2I/T2V the sequence layout is:

<|im_start|> [prompt tokens] <|im_end|> <|vision_start|> [N latent placeholders] <|vision_end|>
                                                          └──── routed through moe_gen ────┘
                                                          ↑ everything else: normal weights

The MLX port (lance_mlx/qwen2_navit_mlx.py) routes by slicing the sequence into the latent slab vs the surrounding text, applying the appropriate expert to each slab, and concatenating. mrope position ids continue to flow normally across both slabs (with axis-T/H/W coordinates only varying inside the latent slab).

Conversion source

Converted from bytedance-research/Lance/Lance_3B/* using a local conversion pipeline. Layout transforms:

• —Conv weights: PT (O, I, [T,] H, W) → MLX (O, [T,] H, W, I)
• —Embedding weights: shape preserved
• —lm_head.weight tied to embed_tokens.weight (Qwen default)
• —All *_moe_gen.* keys copied verbatim under the same names

Related ports

A parallel MLX port exists at mlx-community/Lance-3B-Video-bf16 (Apache-2.0). The two checkpoints have been verified numerically equivalent: remapping this repo's F32 weights into their layout and casting to bf16 produces byte-identical pixel output through their pipeline. Use whichever fits your workflow.

License

Apache 2.0, inherited from upstream bytedance-research/Lance. The Wan 2.2 VAE component is also Apache 2.0 from Alibaba's Wan team.

Acknowledgements

• —ByteDance Research — original Lance training + PT release
• —Qwen team — Qwen2.5-VL-3B-Instruct backbone
• —Alibaba Wan team — Wan 2.2 VAE training
• —Apple `mlx` and `mlx-vlm` teams — the underlying frameworks
• —mlx-community Lance porters — parallel bf16 port, numerically cross-checked against this one
• —This MLX port — RockTalk

Citation

@misc{lance_mlx,
  title  = {Lance-3B-MLX — First MLX port of ByteDance's Lance},
  author = {RockTalk},
  year   = {2026},
  url    = {https://huggingface.co/RockTalk/Lance-3B-MLX}
}