representational-alignment/iclr2026-realign-challenge

Overview

This app runs the Re-Align Hackathon leaderboards. Blue Team submissions select model sets and are ranked by mean pairwise CKA. Red Team submissions select stimuli and are ranked by divergence (1 - avg CKA) computed across cached dummy embeddings.

Local run

conda env create -f environment.yml
conda activate iclr2026-challenge
python app.py

Submission formats

Blue Team JSON

{
  "models": [
    {
      "model_name": "vit_base_patch16_224",
      "source": "dummy_cache",
      "model_parameters": null
    },
    {
      "model_name": "resnet50",
      "source": "dummy_cache",
      "model_parameters": null
    }
  ]
}

Red Team JSON

{
  "differentiating_images": [
    {
      "dataset_name": "cifar100",
      "image_identifier": "test/bear/image_0007.png"
    },
    {
      "dataset_name": "imagenet1k",
      "image_identifier": "val/n03445777/ILSVRC2012_val_00003572.JPEG"
    }
  ]
}

Hugging Face datasets (private only)

You can optionally host the model/stimulus selections as a Hugging Face Dataset and paste the dataset link into the app. These datasets must be private (do not publish public submissions).

Blue Team dataset

Expected column:

• —model_names: list of model name strings

Example dataset row:

{
  "model_names": [
    "vit_base_patch16_224", 
    "resnet50", 
    "convnext_base"
  ]
}

Create the dataset (example with datasets + huggingface_hub):

pip install datasets huggingface_hub

from datasets import Dataset
from huggingface_hub import login

login(token="hf_...")  # optional if you already ran `huggingface-cli login`

rows = [
    {"model_names": ["vit_base_patch16_224", "resnet50", "convnext_base"]},
]
ds = Dataset.from_list(rows)
ds.push_to_hub("your-username/blue-team-submission", private=True)

Paste the dataset link into the app:

https://huggingface.co/datasets/your-username/blue-team-submission

Red Team dataset

Expected column:

• —stimulus_id: list of stimulus key strings (the keys shown in the app dropdown)

Example dataset row:

{
  "stimulus_id": [
    "cifar100:test/bear/image_0007.png", 
    "imagenet1k:val/n03445777/ILSVRC2012_val_00003572.JPEG"
  ]
}

Create the dataset:

pip install datasets huggingface_hub

from datasets import Dataset
from huggingface_hub import login

login(token="hf_...")  # optional if you already ran `huggingface-cli login`

rows = [
    {
        "stimulus_id": [
            "cifar100:test/bear/image_0007.png",
            "imagenet1k:val/n03445777/ILSVRC2012_val_00003572.JPEG",
        ]
    },
]
ds = Dataset.from_list(rows)
ds.push_to_hub("your-username/red-team-submission", private=True)

Paste the dataset link into the app:

https://huggingface.co/datasets/your-username/red-team-submission

Access token (for private datasets)

1. 1.Go to your Hugging Face account settings: https://huggingface.co/settings/tokens
2. 2.Create a new token with the Read scope.
3. 3.Copy the token (it starts with hf_...).
4. 4.Paste it into the "HuggingFace access token (optional)" textbox in the app.

Where submissions are stored

Submissions are stored locally in:

• —hackathon-data/blue_submissions.json
• —hackathon-data/red_submissions.json

You can override storage paths with:

• —HACKATHON_DATA_DIR=/your/path
• —HACKATHON_BLUE_DATA_PATH=/your/path/blue.json
• —HACKATHON_RED_DATA_PATH=/your/path/red.json

Cached embeddings

Dummy cached embeddings live in src/hackathon/data.py. Replace get_dummy_model_embeddings() and list_dummy_stimuli() with real data when it is ready.

See docs/evaluation_contract.md for the draft contract and model registry spec for real forward-pass evaluation.

For storage layout and path hygiene conventions, see docs/storage_layout.md.

To enable Modal-backed scoring, set HACKATHON_MODAL_ENABLE=true and provide HACKATHON_MODEL_REGISTRY plus HACKATHON_STIMULI_CATALOG.