EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1007

Retrain bank: WikiText-2 / GPT-2, random halves, seed 1007

This repository contains 100 fully retrained language models, not just scores.

Each model is GPT-2 (gpt2) fine-tuned on a different random 50% (2,328 documents) of the 4,656-document WikiText-2 training set from `EleutherAI/bergson-wikitext-2-4656-chunks`, following the recipe of Bae et al. 2024, Training Data Attribution via Approximate Unrolled Differentiation (App. B.1). retrained/base is trained on the full set with the same seed.

Five banks share the same 100 subsets (subsets.json) and differ only in training seed: 1004, 1005, 1006, 1007, 1008. Averaging query losses over the five seeds gives the ground truth used in the bergson replication.

What is here

Using it

from huggingface_hub import snapshot_download
import pandas as pd

path = snapshot_download("EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1007", repo_type="dataset")

# ground truth: what training on each half did to each query's loss
truth = pd.read_csv(f"{path}/validation.csv")

# score your own method, then correlate its predicted influence against `diff`
# LDS = mean over queries of Spearman(predicted subset sums, measured diff)

The directory also drops straight into a bergson validate step as retrained_dir.

Measured on the five-seed ground truth

Scores were computed on the fine-tuning run in `EleutherAI/bergson-wikitext-2-gpt2`.

Provenance

• —optimizer adamw (β=(0.9, 0.999), ε=1e-8), lr 3e-5 constant, weight decay 0.01, batch size 8, 3 epochs, fp32, dropout on, seed 1007
• —corpus: WikiText-2, 4,656 training documents; 481 validation queries

Produced by bergson (examples/replicate_bae_approx_unrolling_source/wikitext_gpt2_retrain.yaml).