brendanlong/sequential-transformer-lens-experiment
Checkpoints for Training a Transformer to Compose One Step Per Layer (and Proving It) Final checkpoints behind the writeup Training a Transformer to Compose One Step Per Layer (and Proving It). Code, analyses and the full experiment log: github.com/brendanlong/sequential-transformer-lens-experiment. Training curves: public wandb project. Every checkpoint is a torch.save dict {"step", "model_state_dict", "model_config"} loadable with torch.load(..., weights_only=True); each has a… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/sequential-transformer-lens-experiment.
Checkpoints for Training a Transformer to Compose One Step Per Layer (and Proving It)
Final checkpoints behind the writeup Training a Transformer to Compose One Step Per Layer (and Proving It). Code, analyses and the full experiment log: github.com/brendanlong/sequential-transformer-lens-experiment. Training curves: public wandb project.
Every checkpoint is a torch.save dict {"step", "model_state_dict", "model_config"} loadable with torch.load(..., weights_only=True); each has a .json sidecar with a description, the wandb run, and a sha256. Load one with the repo's lego.training.load_model("hf:lego/<run>/<file>").
Each sidecar's source_artifact is the wandb artifact the bytes were copied from. The research code named artifacts after the script, not the run, so several runs share a collection name and are told apart only by version (e.g. every _FSL run here came from lego-std_96d_6h_8L_curriculum_FSL:vN, and the weight-shared runs' collections say 8L where the run names say 8iter); the sha256, wandb_run and description are the authoritative identity.
Layout
lego/<run_name>/step_<N>.pt # checkpoint
lego/<run_name>/step_<N>.pt.json # description, wandb run, metadata, sha256All models are 96-dim, 6-head, 8-layer decoder-only transformers on S3 (≅ D3) group composition with chain lengths k = 0–6; kp2 = training data weighted by k², uniform = uniform over k; curriculum_AO / curriculum_FSL = after phase 1 (answer-only loss) / phase 2 (full-sequence loss) of the AO→FSL curriculum; fsl_only = full-sequence loss from scratch; ws_…_8iter = weight-shared (universal) transformer with one block looped 8 times; s<seed>.