ddebree/JEPA-demo

JEPA-demo

Streamlit demo for probing frozen I-JEPA visual representations against YOLO-format obstacle labels.

This repository is designed to run locally and as a Docker-based Hugging Face Space.

Concept

This project explores representation-first operational vision:

• —YOLO provides object-level grounding: labels, boxes, and counts.
• —I-JEPA provides frozen visual representations: saliency, context, and scene structure.
• —A tiny logistic-regression classifier tests whether frozen I-JEPA embeddings are already enough to classify object crops.

The classifier is intentionally small. Its trainable parameter count is approximately:

embedding_dim * num_classes + num_classes

For example, with a 1,280-dimensional I-JEPA embedding and 10 classes, the head has about 12,810 trainable parameters while I-JEPA remains frozen.

Logistic Regression Head

The lightweight classifier is logistic regression, not linear regression.

Linear regression predicts a continuous value:

features -> linear formula -> number

Logistic regression uses a linear formula too, but converts class scores into probabilities:

I-JEPA crop embedding
        |
linear scores per class
        |
softmax probabilities
        |
predicted class

In this project, that means the large I-JEPA model provides the visual embedding and stays frozen. The tiny logistic-regression head only learns simple linear decision boundaries between YOLO classes in that embedding space.

The goal is not only object detection. The goal is to inspect whether a representation model can support broader scene understanding: isolated objects, group-like scenes, context-heavy surroundings, and multi-region visual structure.

Defaults

Features

• —Loads YOLO-format obstacle metadata from a Hugging Face dataset.
• —Downloads source images from the dataset repository.
• —Runs frozen I-JEPA image and patch embeddings.
• —Displays YOLO boxes as benchmark labels.
• —Overlays I-JEPA patch saliency on the image.
• —Estimates scene structure from connected saliency regions.
• —Compares YOLO labels with I-JEPA class prototypes built from reference images.
• —Optionally trains a tiny LogisticRegression classifier on frozen I-JEPA crop embeddings.
• —Analyzes object context using object crop, context crop, and scene embeddings.

Local Setup

cd ~/projects/JEPA-demo
uv sync

Run the app:

uv run streamlit run app.py

Open the Streamlit URL, usually:

http://localhost:8501

The first run downloads the I-JEPA checkpoint and dataset images.

Bulk Evaluation

For larger offline runs, use:

uv run python -m src.bulk_eval \
  --eval-samples 50 \
  --support-samples 80

--support-samples is used for both class prototypes and tiny-classifier training. --eval-samples is the number of images evaluated afterward. Support and eval images are kept disjoint by file name.

Outputs:

outputs/bulk_eval.csv
outputs/run_YYYYMMDD_HHMMSS/objects.csv
outputs/run_YYYYMMDD_HHMMSS/summary.csv
outputs/run_YYYYMMDD_HHMMSS/per_class.csv
outputs/run_YYYYMMDD_HHMMSS/prototype_confusion.csv
outputs/run_YYYYMMDD_HHMMSS/head_confusion.csv
outputs/run_YYYYMMDD_HHMMSS/report.json

By default, bulk runs create a timestamped run directory under outputs/. The main CSV also includes run_id, support_samples, eval_samples, and seed columns. The summary files include overall accuracy, macro accuracy, per-class accuracy, confusion matrices, and a compact JSON report with top confusions and file references.

Hugging Face Spaces

Use a Docker Space.

The included Dockerfile:

• —installs uv
• —runs uv sync --locked
• —exposes port 7860
• —starts Streamlit on 0.0.0.0:7860

Method

YOLO dataset metadata
        |
        v
download image
        |
        v
frozen I-JEPA
        |
        +--> patch embeddings -> saliency overlay + scene structure
        |
        +--> object crop embeddings -> prototype match
        |
        +--> object/context/scene embeddings -> context pattern
        |
        `--> optional LogisticRegression head -> class prediction

UI Signals

• —Scene structure agreement: whether I-JEPA saliency structure matches YOLO single-vs-multiple structure.
• —Prototype label agreement: whether nearest class prototype matches the YOLO object label.
• —Small head agreement: whether the optional lightweight classifier matches the YOLO object label.
• —Context pattern: embedding-based estimate of object surroundings.
• —Context strength: low/medium/high similarity signal between object, context, and scene embeddings.

Signal terms:

• —isolated / object-dominant: one object is the main visual focus.
• —group / crowd context: the object appears in a broader group-like situation.
• —near other objects / scene-embedded: the object and surrounding scene are strongly related.
• —multi-region: several separate salient visual regions.
• —distributed/group pattern: saliency is spread across a broader clustered scene.

Notes

I-JEPA is not used as a trained detector in this demo. YOLO labels are the benchmark reference. I-JEPA is used as a frozen representation model for saliency, class-prototype matching, context analysis, and lightweight supervised heads.

Prototype quality depends on support coverage. Rare or visually specific classes may require more support samples before prototype matching becomes stable.

Project Layout

.
|-- app.py
|-- Dockerfile
|-- pyproject.toml
|-- uv.lock
|-- README.md
|-- docs/
|   `-- blog_outline.md
`-- src/
    |-- __init__.py
    |-- benchmark_similarity.py
    |-- bulk_eval.py
    |-- context_analysis.py
    |-- extract_embeddings.py
    |-- ijepa_localization.py
    |-- jepa_adapter.py
    |-- obstacle_dataset.py
    |-- prototypes.py
    |-- small_head.py
    |-- utils.py
    `-- visualization.py