Modelpublic

businesslion/scanpath-prediction-checkpoints

sourceHugging Facemitupdated 25d agoView on Hugging Face

0likes

Model Card

Scanpath Prediction Checkpoints

Model checkpoints for a transformer-based scanpath prediction architecture — predicting the sequence of eye fixations (x, y) a human makes when free-viewing an image.

Evaluated against DeepGaze III (Kümmerer, Bethge & Wallis, Journal of Vision 2022) on MIT1003 and CAT2000.

⚠️ Which folder should I use?

Use `v2_phase2/`. It holds the current, published configuration. The nb17/, nb19/, nb20/ and nb21_* folders are older exploratory runs kept for provenance — they use different hyperparameters (K=4 instead of K=8, plain L2 loss, learned-sigma variant) and will not reproduce the reported numbers.

Results (`v2_phase2`)

L2 = mean Euclidean distance between predicted and ground-truth fixation, with each axis normalized independently to [0,1] (x/W, y/H). Teacher-forced, all valid steps 1..8, 10-fold cross-validation on predefined 8/1/1 splits (identical splits to DeepGaze III).

Dataset	Ours	DeepGaze III	Improvement
MIT1003	0.1227	0.2112	−42%
CAT2000	0.1432	0.1998 *	−28%

\ DeepGaze III on CAT2000 is a transfer* baseline — its scanpath components are cross-validated on MIT1003 and were never retrained on CAT2000. We ensemble all 10 components. Label it as transfer wherever the number is used.

Per-step L2 (per-axis, [0,1])

step	MIT1003 ours	MIT1003 DG3	CAT2000 ours	CAT2000 DG3
1	0.1220	0.139	0.1051	0.168
2	0.1211	0.204	0.1176	0.175
3	0.1167	0.223	0.1389	0.172
4	0.1184	0.224	0.1510	0.176
5	0.1256	0.224	0.1576	0.184
6	0.1247	0.230	0.1583	0.187
7	0.1271	0.231	0.1591	0.189
8	0.1261	0.229	0.1606	0.188

Our model has lower error at every fixation step on both datasets.

v2_phase2/
  mit1003/
    dinov2_big_unfrozen/     fold0..fold9  (+ trainsummary json)   best L2 on MIT1003
    clip_small_unfrozen/     fold0..fold9  (+ trainsummary json)   best NSS on MIT1003
  cat2000/
    dinov2_big_unfrozen/     fold0..fold9  (+ trainsummary json)

nb17_mit1003_cv10/               older baseline (pre-Phase-2), single-fold samples
nb19_mit1003_cv10_k4/            K_FIX=4 ablation
nb20_cat2000_cv10_k4/            K_FIX=4 ablation, CAT2000
nb21_mit1003_cv10_learned_sigma/ learned-sigma experiment

Each v2_phase2 config ships all 10 CV folds, so the cross-validated numbers above are fully reproducible. trainsummary_*.json records per-fold best val-NSS, per-epoch history and wall-clock time.

Architecture

image → encoder (frozen or fine-tuned) → transformer autoregressive decoder → Linear(d, 2) → (x, y)

Config	Encoder	enc_dim	dec_dim	layers	batch
`dinov2_big`	DINOv2 ViT-B/14	512	512	10	32
`clip_small`	CLIP	512	256	4	128

—K_FIX = 8 fixations predicted per image
—context_len = 4 — the decoder's causal attention sees at most the last 4 fixations, matching DeepGaze III's history length
—First fixation is predicted from a BOS token at the image centre (0.5, 0.5)

Training hyperparameters

image size    224 x 224          (448 was tested and was worse)
epochs        30
optimiser     AdamW (fused), weight_decay 5e-4
lr            3.3e-4 head; 2e-5 encoder when unfrozen  (differential LR)
schedule      3-epoch linear warmup -> cosine
precision     bfloat16 AMP, drop_last=True
dropout       0.1
grad clip     max_norm 1.0
augmentation  horizontal flip (train only), x -> 1-x
loss          masked_L2 + 0.3 * NSS_aux  (map_size 128, sigmas 3/5/8, active from epoch 3)
selection     best checkpoint by validation NSS (not validation L2)
seed          42
CV            10 folds, 8/1/1, predefined splits (identical to DeepGaze III)

Usage

python

import torch
from huggingface_hub import hf_hub_download
from scanpath_lib.models.scanpath_model import ScanpathModel

path = hf_hub_download(
    "businesslion/scanpath-prediction-checkpoints",
    "v2_phase2/mit1003/dinov2_big_unfrozen/fold0_dinov2_big_unfrozen.pt",
)

model = ScanpathModel(
    k_fix=8, enc_dim=512, dec_dim=512, n_heads=4, n_layers=10,
    max_len=64, encoder_name="dinov2", image_size=224,
    context_len=4, predict_sigma=False,
)
model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
model.eval()

For the cross-validated numbers, evaluate each fold's checkpoint on that fold's test split (every image is then scored exactly once by a model that never trained on it) and pool.

Reproducibility note

The v2_phase2 weights were retrained in Aug 2026 after the originals were lost, using the same code, seed and fold splits but a newer PyTorch (2.10). They reproduce the original numbers to within 0.7% (MIT1003 0.1219 → 0.1227; CAT2000 0.1426 → 0.1432). Exact bit-level equality is not expected across PyTorch versions.

Citation

DeepGaze III baseline:

bibtex

@article{kummerer2022deepgaze3,
  title   = {DeepGaze III: Modeling free-viewing human scanpaths with deep learning},
  author  = {K{\"u}mmerer, Matthias and Bethge, Matthias and Wallis, Thomas S. A.},
  journal = {Journal of Vision},
  volume  = {22}, number = {5}, pages = {7}, year = {2022},
  doi     = {10.1167/jov.22.5.7}
}