anonymousML123/factorjepa-lpft-vjepa21-vitG-2B-poc

factorjepa-lpft-vjepa21-vitG-2B-poc

FactorJEPA — V-JEPA 2.1 ViT-G (2B) adapted on Indian-context urban driving / walking / monument clips.

This is the `lpft` arm of the iter18 FactorJEPA ablation: LP-FT baseline: linear-probe warmup then full fine-tune (Kumar et al. ICLR'22) (B2). The study compares factor-surgery against strong fine-tuning baselines on the claim vjepa_surgery >> vjepa_pretrain >> vjepa_frozen for motion / temporal features on Indian urban video. Every non-pretrain arm initializes from the SAME continual-pretrain checkpoint (fair duel — identical data, identical starting weights).

🎯 Training summary

📈 Training trajectory (initial → final, from probe_history.jsonl)

(5 checkpoints across training.)

🧪 Held-out test evaluation (N=1825 clips · 95% BCa bootstrap CI)

Direction: ↑ higher better · ↓ lower better · `·` signed diagnostic. `—` = not computed for this arm.

⚡ Quick start — self-contained, no other code needed

This repo ships everything: the weights, the architecture (vjepa2_src/, vendored Meta V-JEPA 2 source, MIT), and a loader. Download it and run — no private package, no separate clone.

huggingface-cli download anonymousML123/factorjepa-lpft-vjepa21-vitG-2B-poc --local-dir factorjepa-model
cd factorjepa-model && pip install -r requirements.txt
python load_factorjepa.py --encoder student_encoder.pt    # builds 2B ViT-G, loads, forwards (no video needed)

from load_factorjepa import load_encoder, preprocess_frames, extract_features
encoder = load_encoder("student_encoder.pt", device="cuda")   # bf16 on cuda, fp32 on cpu
clip = preprocess_frames(frames_uint8)[None]                  # (T,H,W,3) uint8 -> (1, 16, 3, 384, 384)
feats = extract_features(encoder, clip)                       # (1, 4608, 1664) token features

NATIVE V-JEPA 2.1 ViT weights — NOT `transformers.VJEPA2Model` (AutoModel.from_pretrained fails: different keys + no 2.1 deep-supervision head). No `xformers` (SDPA attention). student_encoder.pt is encoder-only — for an actual next-frame prediction heatmap also load the predictor from m09c_ckpt_best.pt (key predictor): from load_factorjepa import load_predictor; predictor = load_predictor("m09c_ckpt_best.pt", device="cuda").

🏗️ Architecture

The exact constructor kwargs are in load_factorjepa.py (verified against the eval pipeline that produced these weights). student_encoder.pt wraps the weights under the key student_state_dict — the loader unwraps it, strips module./backbone. prefixes, and asserts ≥90% of params load (fail-loud).

📦 Files in this repo

🧪 Reproducibility

This checkpoint was produced by:

CACHE_POLICY_ALL=2 ./scripts/run_train.sh lpft --FULL \
    2>&1 | tee logs/lpft_full.log

Pipeline source: iter/iter14_surgery_on_pretrain/plan_HIGH_LEVEL.md

📜 Attribution & license — the links below are provenance/credit only, NOT a setup step

✅ 100% self-contained. Everything needed to load this model is already in THIS repo (vjepa2_src/ + load_factorjepa.py + the weights). You do not need to visit, clone, pip install, or download anything from the two links below — they are license/credit only. Loading touches no other repo.

• —Adapted weights (student_encoder.pt, m09*_ckpt_best.pt, motion_aux_head.pt) — Apache-2.0 (this repo). Derived from facebook/v-jepa-2-vitg (provenance only — not needed to load).
• —Vendored architecture (vjepa2_src/) — Meta Platforms' V-JEPA 2, MIT, copied unmodified from github.com/facebookresearch/vjepa2 @ 204698b (credit only — the code is already in `vjepa2_src/`; its MIT license is at `vjepa2_src/LICENSE`). © Meta Platforms, Inc. and affiliates.

📝 Citation

@misc{factorjepa2026,
  title  = {FactorJEPA: Factor-disentangled SSL for Indian-context urban video},
  author = {Wanaskar, Kapil and others},
  year   = {2026},
  note   = {HF model card auto-generated by src/utils/hf_finetuned_push.py}
}

Model card auto-generated by `src/utils/hf_finetuned_push.py` at 2026-06-12T11:11:32+00:00.