vasanth009/vjepa2-vitg-fpc64-256-mlx

V-JEPA2 ViT-g → MLX (TRIBE vision backbone)

MLX (fp16) conversion of [`facebook/vjepa2-vitg-fpc64-256`](https://huggingface.co/facebook/vjepa2-vitg-fpc64-256) for fast video feature extraction on Apple Silicon.

Used by the TRIBE v2 Mac fork: vasanthsreeram/tribev2 with the brain encoding head `facebook/tribev2`.

Files

How this was converted

1. 1.Load HF AutoModel for facebook/vjepa2-vitg-fpc64-256 in fp32.
2. 2.Map tensors with vjepa2-mlx convert_state_dict (Conv3d layout, QKV, etc.).
3. 3.Keep *`encoder.` only** (predictor dropped — not used by TRIBE).
4. 4.Cast to fp16, save safetensors with format=mlx metadata.
5. 5.Config: mlp_ratio=48/11 to match HF (not integer 4).

Full write-up: docs/MLX_CONVERSION.md in the code fork.

Validation

Against official torch TRIBE vision-only preds on a 1 s clip (same config):

• —Cosine ≈ 0.96, Pearson r ≈ 0.96
• —Encoder-only random input vs HF: cosine ≈ 0.9994
• —Runtime: ~11× faster than torch CPU encode on the test Mac

Usage with TRIBE fork

hf download vasanth009/vjepa2-vitg-fpc64-256-mlx --local-dir mlx_weights/V-JEPA2-vitg-fpc64-256
# then in the tribev2 fork:
python demo_data/run_efficient.py your.mp4 --open

from tribev2.mlx_vjepa import encode_clip_frames, install_mlx_video_hooks
# requires vjepa2_mlx package for the encoder graph

License & attribution

• —Weights derived from Meta V-JEPA2 — CC-BY-NC-4.0 (non-commercial), same spirit as upstream.
• —Cite the TRIBE paper if you use the brain model.
• —Conversion utilities based on vjepa2-mlx.

Not affiliated with Meta. This is a community conversion for Apple Silicon inference.