CoolFace
Datasetpublic

EleutherAI/bergson-wikitext-gpt2-leaderboard-bank

bergson leaderboard: retrain banks, scores and LDS/QLD results (WikiText GPT-2) Everything behind the numbers on the bergson leaderboard, for the model at EleutherAI/bergson-wikitext-gpt2-leaderboard. path what it is bank/ the LDS ground truth: 100 random leave-1%-out subsets of the 4,608 training chunks (subsets.json) and each subset's measured loss change on the 50 test queries (validation.csv) random/retrained/{base,subset_0..99} the retrained models themselves… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/bergson-wikitext-gpt2-leaderboard-bank.

sourceHugging Faceapache-2.0updated 12d agoView on Hugging Face
0likes183downloads
Dataset Card

bergson leaderboard: retrain banks, scores and LDS/QLD results (WikiText GPT-2)

Everything behind the numbers on the bergson leaderboard, for the model at `EleutherAI/bergson-wikitext-gpt2-leaderboard`.

pathwhat it is
bank/the LDS ground truth: 100 random leave-1%-out subsets of the 4,608 training chunks (subsets.json) and each subset's measured loss change on the 50 test queries (validation.csv)
random/retrained/{base,subset_0..99}the retrained models themselves and the same bank's validation.csv, subsets.json
magic_seed43/the same 100 subsets retrained at seed 43 (the MAGIC cross-seed row): validation.csv, summary.csv
scores/<method>/one bergson score store per leaderboard row: scores.bin + info.json, 4,608 training chunks x 50 queries
lds/lds_<method>.jsonper-query LDS against bank/ with a 10k bootstrap over subsets (lds_from_bank.py)
qld/qld_<method>.json, filters/filter_<method>/proponent-filter results: retrain without each query's top 1% and the query loss difference vs random removal (qld_from_filters.py)

Reproducing

Run the configs in `examples/compare_wikitext` in the order its README gives: 1_magic.yaml (train + bank), 2_interval.yaml (checkpoints), one yaml per method, filters/*.yaml, then lds_from_bank.py, qld_from_filters.py and lds_tables.py.

Model: GPT-2 fine-tuned 4 epochs on `EleutherAI/bergson-wikitext-512-chunks` (AdamW, betas 0.9/0.999, eps_root 1e-17, lr 4e-4, batch 256, seed 42).

Leaderboard

Linear datamodeling score (LDS) is the accuracy of a method for producing global data rankings by influence. The query loss difference (QLD) shows how much model loss for a held-out query can be increased by retraining without the most highly ranked data by influence (here the top 1%), compared to a random removal baseline.

MethodProponent QLD [95% CI]LDS [95% CI]
MAGIC0.100 [0.090, 0.112]0.931 [0.925, 0.936]
Eigenvalue-corrected Shampoo0.071 [0.060, 0.082]0.517 [0.491, 0.539]
EK-FAC0.070 [0.058, 0.082]0.454 [0.426, 0.479]
KFAC0.067 [0.056, 0.080]0.420 [0.391, 0.446]
BM250.062 [0.048, 0.076]0.220 [0.185, 0.252]
Qwen3-Embedding-8B semantic search0.049 [0.038, 0.061]0.132 [0.093, 0.169]
TrackStar (no optimizer correction, projection 64)0.045 [0.036, 0.055]0.270 [0.240, 0.295]
TRAK (8-model ensemble)0.032 [0.024, 0.040]0.138 [0.111, 0.165]
SOURCE (Adam)0.024 [0.018, 0.030]0.154 [0.126, 0.181]
Gradient cosine similarity0.021 [0.016, 0.027]0.156 [0.131, 0.181]
Activation similarity0.000 [-0.000, 0.001]0.110 [0.070, 0.149]

Part of the Data Attribution collection.