YenugulaAIML/scaleop-pythia-conversions

ScaleOp — Pythia size-conversion checkpoints

Research artifacts for the paper "Wiring Beats Blending: What Transfers Between Transformer Sizes — and What Doesn't" (arXiv:2608.02829).

The paper asks whether a pretrained large model can be converted into a smaller sibling instead of training the small size from scratch. These checkpoints are the endpoints of matched-budget recovery races: every arm gets the same target shape, data order, schedule, and token budget, and only the starting weights differ. The initializations compared:

What's in this repo

All files are plain PyTorch state_dicts for GPTNeoXForCausalLM with the target size's stock Pythia config. They are research checkpoints trained for 30M–1B tokens (0.01–0.3% of Pythia's pretraining), useful for studying conversion initializations — not production or instruction-tuned models.

Pair A: Pythia-1.4B → 410M

Reference points: the real Pythia-410M scores ≈15.6 on the same eval; transfer initialization beats from-scratch by up to 18× at 30M tokens, and both rescale arms converge to parity at 1B.

Pair B: Pythia-410M → 160M (depth-dominated, 3 data-draw seeds)

Per-seed numbers and the full ablation ladder are in the paper's appendix.

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoConfig, AutoTokenizer, GPTNeoXForCausalLM

path = hf_hub_download(
    "YenugulaAIML/scaleop-pythia-conversions",
    "pairA_1.4b-to-410m/hybrid_rs_1B.pt",
)
config = AutoConfig.from_pretrained("EleutherAI/pythia-410m")   # target size's stock config
model = GPTNeoXForCausalLM(config)
model.load_state_dict(torch.load(path, map_location="cpu"))
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-410m")

out = model.generate(**tokenizer("The capital of France is", return_tensors="pt"), max_new_tokens=10)
print(tokenizer.decode(out[0]))

For pair B, use the EleutherAI/pythia-160m config and tokenizer instead.

Training data

Continued pre-training on a deterministic stream of pile-uncopyrighted, with a frozen 10,000×128 evaluation corpus that is contamination-guarded against the training stream. The full recipe, configs, and seeds are in the code repository.

Limitations

These are controlled-experiment artifacts, not usable assistants: budgets stop far short of full pretraining, all models are base LMs with no instruction tuning or safety training, and they inherit the biases of the Pile and the Pythia family. Single-seed cells are noted in the paper.

Links

• —Paper: arXiv:2608.02829
• —Code, configs, and the frozen-corpus recipe: github.com/rsdpyenugula/ScaleOp

Citation

@article{yenugula2026wiring,
  title   = {Wiring Beats Blending: What Transfers Between Transformer Sizes --- and What Doesn't},
  author  = {Yenugula, Ravi Satya Durga Prasad},
  journal = {arXiv preprint arXiv:2608.02829},
  year    = {2026}
}