aavi21458/vjepa2-latent-prediction

V-JEPA2 Latent Prediction

Most vision systems identify objects. JEPA identifies behaviors - short temporal processes such as lifting, sliding, placing, or retrying.

V-JEPA2 learns by predicting in latent space rather than reconstructing pixels, capturing temporal abstractions and action primitives that depend on direction, sequence, or intent rather than appearance.

Demo Description

This demo uses a frozen V-JEPA2 encoder with a lightweight classifier trained on Something-Something-V2 (SSV2) to extract temporal action primitives from short video segments (~3s).

The classifier distinguishes subtle temporal variations such as:

• —push left vs push right
• —open → close
• —lift → slide → place
• —fail → retry

These differ primarily in direction and order, illustrating temporal structure in latent space.

Model Details

• —Backbone: V-JEPA2 ViT-L (frozen)
• —Head: SSV2 classifier (174 temporal classes)
• —Input: unlabeled video (≤30s)
• —Inference: sliding windows (~3s)
• —Objective: temporal abstraction without pixel reconstruction

The encoder remains frozen to preserve the temporal representations learned during self-supervised pretraining.

Motivation

Understanding processes rather than objects is central to:

• —manipulation
• —tool use
• —planning and control
• —embodied and interactive agents

JEPA provides these abstractions via latent prediction, making it a candidate foundation for embodied AI.

No pixel reconstruction. No generative decoder. Just temporal structure.

References

Paper: https://arxiv.org/abs/2506.09985 Code: https://github.com/facebookresearch/vjepa2 Model: https://huggingface.co/facebook/vjepa2-vitl-fpc16-256-ssv2