Datasetpublic

EleutherAI/LDS-retrain-bank-adamw-N16k-bs256

Retrain bank: sm_adamw_eps1e17_16k_bs256 This repository contains 100 fully retrained language models, not just scores. Each model is GPT-2 (gpt2) fine-tuned on the same 16,000-document corpus with a different random 1% (160 documents) held out, from the same seed and the same data order as the base model in retrained/base. Retraining is deterministic within one environment, so the models differ only by the documents removed. That is the expensive part of any leave-k-out… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/LDS-retrain-bank-adamw-N16k-bs256.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes665downloads

Dataset Card

Retrain bank: `sm_adamw_eps1e17_16k_bs256`

This repository contains 100 fully retrained language models, not just scores.

Each model is GPT-2 (gpt2) fine-tuned on the same 16,000-document corpus with a different random 1% (160 documents) held out, from the same seed and the same data order as the base model in retrained/base. Retraining is deterministic within one environment, so the models differ only by the documents removed.

That is the expensive part of any leave-k-out attribution study, and it is reusable: a new attribution method can be evaluated against this bank without retraining anything.

What is here

path	what it is
`retrained/base/`	the unablated fine-tuned model
`retrained/subset_*/`	100 models, each missing a different 1% of the corpus
`validation.csv`	the ground truth: per (subset, query) change in loss caused by that removal
`subsets.json`	which document ids each subset removed
`config.yaml`	the exact training configuration
`filter_proponents_*/`	tail-filter results: loss change when a scorer's top-ranked 1% is removed

Using it

python

from huggingface_hub import snapshot_download
import pandas as pd

path = snapshot_download("EleutherAI/metasmoothness-bank-sm_adamw_eps1e17_16k_bs256", repo_type="dataset")

# ground truth: what removing each subset did to each query's loss
truth = pd.read_csv(f"{path}/validation.csv")

# score your own method, then correlate its predicted influence against `diff`
# LDS = mean over queries of Spearman(predicted subset sums, measured diff)

Measured on this bank

metric	value
MAGIC LDS	0.9411
EK-FAC LDS	0.4253
metasmoothness	0.9930
tail-filter delta, MAGIC	0.09090 nats
tail-filter delta, EK-FAC	0.05288 nats
tail-filter delta, random control	0.00023 nats

LDS is the mean per-query Spearman correlation between a scorer's predicted subset influence and the measured diff. The tail-filter delta is a different question on the same bank: remove the 1% a scorer ranks most influential, retrain once, and measure the query loss change against the bank's random removals as the matched control.

Provenance

—optimizer adamw, lr 0.0002, batch size 256, 2 epochs, 125 steps, seed 42
—corpus: smollm2, 16,000 documents
—retrains for one bank all run on a single GPU type: mixing types changes the retrained models by enough to shift LDS by ~0.05, which is larger than most effects being measured.

Produced by bergson.