Modelpublic

alignment-decision-lab/robustness-model-bank

sourceHugging Facemitupdated 13h agoView on Hugging Face

0likes6downloads

Model Card

The Pile Information

This model bank is trained on domains from The Pile. The original Pile release has been largely taken down / restricted (mainly over the Books3 component), so not all 22 official components are available today. Of the 22, working data sources exist for 18; the remaining 4 have no confirmed source anywhere in this project.

*Tier 1 -- true exact-subset mirrors (`timaeus/pile-`):**

Component	Source
Pile-CC	`timaeus/pile-pile-cc`
Github	`timaeus/pile-github`
PubMed Abstracts	`timaeus/pile-pubmed_abstracts`
DM Mathematics	`timaeus/pile-dm_mathematics`
FreeLaw	`timaeus/pile-freelaw`
ArXiv	`timaeus/pile-arxiv`
HackerNews	`timaeus/pile-hackernews`
Enron Emails	`timaeus/pile-enron_emails`

Tier 2 -- other confirmed working sources (proxy / close-alternative quality):

Component	Source
PubMed Central	`datajuicer/the-pile-pubmed-central-refined-by-data-juicer`
OpenWebText2	`suolyer/pile_openwebtext2`
Stack Exchange	`flax-sentence-embeddings/stackexchange_title_body_jsonl`
USPTO Backgrounds	`common-pile/uspto_filtered`
Gutenberg (PG-19)	`emozilla/pg19`
Wikipedia (en)	`wikimedia/wikipedia`
Ubuntu IRC	`common-pile/ubuntu_irc`
BookCorpus2	`Yuti/bookcorpus` (approximate -- likely plain BookCorpus, not confirmed to be specifically "2")
EuroParl	`Helsinki-NLP/europarl`
YoutubeSubtitles	`suolyer/pile_youtubesubtitles`

Not available (no source found in this project): Books3, OpenSubtitles, PhilPapers, NIH ExPorter.

The 3 sources used for the KL-DRO model bank below (FreeLaw, PubMed Central, ArXiv) are all Tier 1 or a vetted close alternative.

Robustness Model Bank

KL-DRO-trained checkpoints across model sizes, source datasets, and robustness coefficients, used by diagnostic_experiment/algorithm_2.py and diagnostic_experiment/hierarchical_routing.py for shift-aware model selection and interpolation.

Status: 23/27 gpt2-medium checkpoints trained. See model_bank_metadata.csv for the full grid and per-checkpoint status. Other model sizes (gpt2Tiny/, gpt2Small/, gpt2Large/, gpt2Xlarge/) are placeholders -- no checkpoints trained yet.

Layout

gpt2Medium/          <- gpt2-medium, 23/27 trained
gpt2Tiny/            <- not started
gpt2Small/           <- not started
gpt2Large/           <- not started
gpt2Xlarge/          <- not started

gpt2-medium

Dataset	Lambda	Path	Status
FreeLaw	lambda_0	`gpt2Medium/FreeLaw/lambda_0/`	trained
FreeLaw	lambda_0.05	`gpt2Medium/FreeLaw/lambda_0.05/`	trained
FreeLaw	lambda_0.1	`gpt2Medium/FreeLaw/lambda_0.1/`	trained
FreeLaw	lambda_0.2	`gpt2Medium/FreeLaw/lambda_0.2/`	trained
FreeLaw	lambda_0.3	`gpt2Medium/FreeLaw/lambda_0.3/`	pending
FreeLaw	lambda_0.4	`gpt2Medium/FreeLaw/lambda_0.4/`	pending
FreeLaw	lambda_0.5	`gpt2Medium/FreeLaw/lambda_0.5/`	trained
FreeLaw	lambda_0.7	`gpt2Medium/FreeLaw/lambda_0.7/`	trained
FreeLaw	lambda_1	`gpt2Medium/FreeLaw/lambda_1/`	trained
PubMed Central	lambda_0	`gpt2Medium/PubMed_Central/lambda_0/`	trained
PubMed Central	lambda_0.05	`gpt2Medium/PubMed_Central/lambda_0.05/`	trained
PubMed Central	lambda_0.1	`gpt2Medium/PubMed_Central/lambda_0.1/`	trained
PubMed Central	lambda_0.2	`gpt2Medium/PubMed_Central/lambda_0.2/`	trained
PubMed Central	lambda_0.3	`gpt2Medium/PubMed_Central/lambda_0.3/`	pending
PubMed Central	lambda_0.4	`gpt2Medium/PubMed_Central/lambda_0.4/`	pending
PubMed Central	lambda_0.5	`gpt2Medium/PubMed_Central/lambda_0.5/`	trained
PubMed Central	lambda_0.7	`gpt2Medium/PubMed_Central/lambda_0.7/`	trained
PubMed Central	lambda_1	`gpt2Medium/PubMed_Central/lambda_1/`	trained
ArXiv	lambda_0	`gpt2Medium/ArXiv/lambda_0/`	trained
ArXiv	lambda_0.05	`gpt2Medium/ArXiv/lambda_0.05/`	trained
ArXiv	lambda_0.1	`gpt2Medium/ArXiv/lambda_0.1/`	trained
ArXiv	lambda_0.2	`gpt2Medium/ArXiv/lambda_0.2/`	trained
ArXiv	lambda_0.3	`gpt2Medium/ArXiv/lambda_0.3/`	trained
ArXiv	lambda_0.4	`gpt2Medium/ArXiv/lambda_0.4/`	trained
ArXiv	lambda_0.5	`gpt2Medium/ArXiv/lambda_0.5/`	trained
ArXiv	lambda_0.7	`gpt2Medium/ArXiv/lambda_0.7/`	trained
ArXiv	lambda_1	`gpt2Medium/ArXiv/lambda_1/`	trained

Load a specific checkpoint:

python

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "alignment-decision-lab/robustness-model-bank",
    subfolder="gpt2Medium/<dataset>/<lambda_dir>",
)

Produced by: diagnostic_experiment/models_bank.py, config: configs/diagnostic/models_bank.yaml.