CoolFace
Modelpublic

sashaboguraev/pythia-160m-ppt-control_music_steps500-seed1024

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes
Model Card

Pythia-160M Pre-Pretraining: control_music_steps500 (seed 1024)

Trained from scratch using the ppt pre-pretraining research framework.

Training Details

ParameterValue
Base architectureEleutherAI/pythia-160m (reinitialized)
Regimencontrol_music_steps500
Seed1024
Stage 1 datasetShuffled MIDI tokens (unstructured control)
Stage 1 steps500
Stage 2 datasetOpenWebText
Stage 2 steps10000
OptimizerAdamW (lr=1e-3, wd=0.0)
Effective batch size64
Sequence length2048

Control Design

Stage 1: MIDI tokens with shuffled order. Same token distribution as music but no sequential structure. This is the unstructured control.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-control_music_steps500-seed1024")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-control_music_steps500-seed1024")

Citation

If you use this model, please cite the original pre-pretraining papers:

  • —Papadimitriou & Jurafsky (2020) — tilt-transfer
  • —Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
  • —Lee et al. (2024) — NCA pre-pretraining (danihyunlee)