Modelpublic

anonymousML123/factorjepa-surgery-3stage-DI-head-vjepa21-vitG-2B-poc

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes

Model Card

factorjepa-surgery-3stage-DI-head-vjepa21-vitG-2B-poc

FactorJEPA — V-JEPA 2.1 ViT-G (2B) adapted on Indian-context urban driving / walking / monument clips.

This is the `surgery_3stage_DI_head` arm of the iter18 FactorJEPA ablation: surgery HEAD variant: encoder+predictor FROZEN at pretrain init; only the motionaux head trains (3-stage DI mixture). The study compares factor-surgery against strong fine-tuning baselines on the claim `vjepasurgery >> vjepapretrain >> vjepafrozen` for motion / temporal features on Indian urban video. Every non-pretrain arm initializes from the SAME continual-pretrain checkpoint (fair duel — identical data, identical starting weights).

🎯 Training summary

Field	Value
Base model	`facebook/v-jepa-2-vitg`
Stage	`surgery_3stage_DI_head`
Architecture	V-JEPA 2.1 ViT-G (~2B params, 1664-dim, 48 layers, hierarchical concat 6656-dim)
Training data	Indian-context urban clips (10k POC pool, leakage-safe train/val/test split)
Mode	poc
Max epochs	2
Total steps	484
Batch size	32
Train / val clips	7,724 / 451
Best epoch (head val-loss)	1 (loss 1.9680)
Trainable head params	435,490
Mode mixture (L/A/I)	{"L": 0.15, "A": 0.15, "I": 0.7}
Wall time	1.5 h
Encoder	FROZEN at pretrain init (head-only training)

📈 Training trajectory (initial → final, from `probe_history.jsonl`)

Metric	Description	Initial	Final	Δ
`probe_top1`	motion-flow 16-class probe top-1	0.273	0.275	+0.8% 📈
`motion_cos`	intra-vs-inter motion cosine	0.1426	0.1501	+5.3% 📈
`val_jepa_loss`	validation JEPA loss (L1)	0.2130	0.1968	-7.6% 📉
`future_l1`	future-frame L1 (per clip)	0.5369	0.5369	+0.0% ➡️
`block_drift_mean`	mean per-block weight drift	0.68002	0.98912	+45.5% 📈

(4 checkpoints across training.)

🧪 Held-out test evaluation (N=1825 clips · 95% BCa bootstrap CI)

Metric	Value	95% CI (±)	better
action top-1	0.4997	0.0230	↑
taxonomy F1	0.7875	—	↑
motion-cos margin	0.1274	0.0045	↑
future-frame MSE	0.5331	0.0009	↓
rollout drift	0.0077	0.0002	↓
causal L1	0.5583	0.0013	↓
t-dist error	0.0065	0.0003	↓
mask-ratio slope	0.0625	0.0005	↓
order sensitivity	0.0154	0.0012	·
teacher-free drift	0.0377	0.0005	↓

Direction: ↑ higher better · ↓ lower better · `·` signed diagnostic. `—` = not computed for this arm.

⚡ Quick start — self-contained, no other code needed

This repo ships everything: the weights, the architecture (vjepa2_src/, vendored Meta V-JEPA 2 source, MIT), and a loader. Download it and run — no private package, no separate clone.

bash

huggingface-cli download anonymousML123/factorjepa-surgery-3stage-DI-head-vjepa21-vitG-2B-poc --local-dir factorjepa-model
cd factorjepa-model && pip install -r requirements.txt
python load_factorjepa.py --encoder student_encoder.pt    # builds 2B ViT-G, loads, forwards (no video needed)

python

from load_factorjepa import load_encoder, preprocess_frames, extract_features
encoder = load_encoder("student_encoder.pt", device="cuda")   # bf16 on cuda, fp32 on cpu
clip = preprocess_frames(frames_uint8)[None]                  # (T,H,W,3) uint8 -> (1, 16, 3, 384, 384)
feats = extract_features(encoder, clip)                       # (1, 4608, 1664) token features

NATIVE V-JEPA 2.1 ViT weights — NOT `transformers.VJEPA2Model` (AutoModel.from_pretrained fails: different keys + no 2.1 deep-supervision head). No `xformers` (SDPA attention). student_encoder.pt is encoder-only — for an actual next-frame prediction heatmap also load the predictor from m09c_ckpt_best.pt (key predictor): from load_factorjepa import load_predictor; predictor = load_predictor("m09c_ckpt_best.pt", device="cuda").

🏗️ Architecture


Encoder	V-JEPA 2.1 ViT-G — `embed_dim=1664`, `depth=48`, `num_heads=26`, RoPE, 2B params (1.84B exact)
Input	`(B, 3, T=16, 384, 384)` — 16 frames, 384² center-crop, ImageNet-normalized; patch 16, tubelet 2
Tokens	`8 × 24 × 24 = 4608` tokens × 1664-dim (final layer); deep-supervision concat = 4608 × 6656
Predictor	2.1 predictor — `predictor_embed_dim=384`, `depth=24`, `num_heads=12`, dense-loss (`return_all_tokens`)
Attention	`scaled_dot_product_attention` (SDPA) — no xformers

The exact constructor kwargs are in load_factorjepa.py (verified against the eval pipeline that produced these weights). student_encoder.pt wraps the weights under the key student_state_dict — the loader unwraps it, strips module./backbone. prefixes, and asserts ≥90% of params load (fail-loud).

📦 Files in this repo

File	Size	Purpose
`student_encoder.pt`	~7 GB	Inference-ready ViT-G encoder weights (key `student_state_dict`) — load this for features
`m09c_ckpt_best.pt`	~8-14 GB	Best-selected ckpt incl. predictor (key `predictor`) — for next-frame / JEPA prediction
`load_factorjepa.py`	~8 KB	Self-contained loader — build model + load weights + preprocess + forward
`vjepa2_src/`	~100 KB	Vendored V-JEPA 2 architecture (Meta, MIT) — the encoder/predictor classes
`requirements.txt`	<1 KB	Pinned deps (exact versions that load these weights; no xformers)
`motion_aux_head.pt`	~2 MB	Motion auxiliary head (paired with student_encoder)
`training_summary.json`	~2 KB	Final-step metrics
`probe_history.jsonl`	~few KB/step	Per-checkpoint probe + drift metrics
`loss_log.{jsonl,csv}`	~several KB	Per-step JEPA loss trajectory
`.png` / `.pdf`	~few MB	Training trajectory plots (loss, drift, probe trio)

🧪 Reproducibility

This checkpoint was produced by:

bash

CACHE_POLICY_ALL=2 ./scripts/run_train.sh surgery_3stage_DI_head --FULL \
    2>&1 | tee logs/surgery_3stage_DI_head_full.log

Pipeline source: iter/iter14_surgery_on_pretrain/plan_HIGH_LEVEL.md

📜 Attribution & license — the links below are provenance/credit only, NOT a setup step

✅ 100% self-contained. Everything needed to load this model is already in THIS repo (vjepa2_src/ + load_factorjepa.py + the weights). You do not need to visit, clone, pip install, or download anything from the two links below — they are license/credit only. Loading touches no other repo.

—Adapted weights (student_encoder.pt, m09*_ckpt_best.pt, motion_aux_head.pt) — Apache-2.0 (this repo). Derived from facebook/v-jepa-2-vitg (provenance only — not needed to load).
—Vendored architecture (vjepa2_src/) — Meta Platforms' V-JEPA 2, MIT, copied unmodified from github.com/facebookresearch/vjepa2 @ 204698b (credit only — the code is already in `vjepa2_src/`; its MIT license is at `vjepa2_src/LICENSE`). © Meta Platforms, Inc. and affiliates.

📝 Citation

bibtex

@misc{factorjepa2026,
  title  = {FactorJEPA: Factor-disentangled SSL for Indian-context urban video},
  author = {Wanaskar, Kapil and others},
  year   = {2026},
  note   = {HF model card auto-generated by src/utils/hf_finetuned_push.py}
}

Model card auto-generated by `src/utils/hf_finetuned_push.py` at 2026-06-12T11:09:29+00:00.