CoolFace
Datasetpublic

GwendalTsang/repro-incremental-learning-of-sparse-attention-patterns-in-transformers-bundle

Reproduction bundle — Incremental Learning of Sparse Attention Patterns in Transformers ICML 2026 paper #12503 · OpenReview vSRh1qU5sH · arXiv 2602.19143 Everything needed to re-run the reproduction whose results are recorded in the Trackio logbook GwendalTsang/repro-incremental-learning-of-sparse-attention-patterns-in-transformers. Layout upstream/ the authors' official code, vendored unchanged… See the full description on the dataset page: https://huggingface.co/datasets/GwendalTsang/repro-incremental-learning-of-sparse-attention-patterns-in-transformers-bundle.

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes267downloads
Dataset Card

Reproduction bundle — Incremental Learning of Sparse Attention Patterns in Transformers

ICML 2026 paper #12503 · OpenReview `vSRh1qU5sH` · arXiv 2602.19143

Everything needed to re-run the reproduction whose results are recorded in the Trackio logbook GwendalTsang/repro-incremental-learning-of-sparse-attention-patterns-in-transformers.

Layout

upstream/            the authors' official code, vendored unchanged
                     (github.com/ralvarezlucend/IL-SAP-Transformers)
scripts/
  run_transformer.py driver: runs upstream's trainer with a local recorder
                     instead of Weights & Biases
  job_entry.sh       entry point used by the Hugging Face GPU Jobs
  gradient_flow.py   numerical verification of Proposition 1 + Theorems 1-4
  make_figures.py    turns raw run outputs into the logbook figures
  make_poster_embed.py  renders the posterly poster to poster_embed.html
outputs/
  hf_job/            results pulled back from the GPU Jobs
  gradient_flow/     theorem checks + reduced-dynamics trajectories
  figures/           HTML figures + the raw numbers behind each one
poster/              posterly source, gate report, rendered poster

Why a driver instead of run.sh

upstream/run.sh needs a Weights & Biases project (r-alvarezlucendo16/incremental-learning) and Hydra's Ray launcher. scripts/run_transformer.py composes the same Hydra configs, disables W&B, and substitutes a Recorder object for wandb.run, so every scalar the upstream trainer logs lands in metrics.json and every attention snapshot in attention_*.npz. No training or evaluation code was modified — the recorder is installed from the outside.

Rerun

Prerequisites: Python 3.12, torch, hydra-core, wandb (imported but unused), numpy, scipy, pandas, matplotlib, seaborn, plotly.

bash
# Claims 2 + 3 — theory, CPU, ~10 s
python scripts/gradient_flow.py --out outputs/gradient_flow \
    --t-full 1e6 --t-thm1 1e6 --t-thm4 1e6

# Claims 1 + 4 — full single-block transformer (GPU strongly recommended)
python scripts/run_transformer.py --out outputs/full_1layer \
    --experiment full_1layer --steps 3000 --ngram-steps 2000 \
    --train-size 9000 --val-size 3000 --batch-size 3000 \
    --lr 0.003 --attn-every 10 --seed 0 --device cuda

# Claim 5 — dataset-size sweep on the minimal architecture
for N in 100 400 600 2000 3000 6000 9000; do
  python scripts/run_transformer.py --out outputs/minimal_n$N \
      --experiment full_disection --steps 3000 --ngram-steps 2000 \
      --train-size $N --val-size 3000 --batch-size 3000 \
      --lr 0.08 --attn-every 50 --seed 0 --device cuda
done

# figures + tables
python scripts/make_figures.py --full-run outputs/full_1layer \
    --sweep-root outputs --gradient-flow outputs/gradient_flow \
    --out outputs/figures

On Hugging Face Jobs (what was actually used):

bash
hf jobs run --flavor l4x1 --timeout 2h -e RUN=full \
  -v ./:/code:ro -v hf://buckets/<you>/ilsap-repro-artifacts:/data \
  pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime bash /code/scripts/job_entry.sh

RUN selects the experiment group: full (claims 1 + 4), dataset (claim 5), seeds (seed robustness for claim 1).

Conventions worth knowing

  • —Steps 0..1999 train the restricted-context reference transformers (unigram/bigram/trigram = context 4/8/12); the transformer under study trains at steps 2000..2999. All plots subtract 2000, matching the authors' own analysis/utils.py (_step_shift defaults to 2000).
  • —The attention snapshots are already cropped to the predicted positions, so the first w = 12 queries do not contain their full history inside the map; offset_profile() in make_figures.py averages only over queries >= w.
  • —gradient_flow.py integrates the flow with Pi(s)^2 (gradient flow on the softmax logits, as written in Section 4). The authors' Figure-10 notebook uses a single Pi, i.e. mirror flow on the simplex; --pi-power 1 reproduces that and gives the same stage ordering.