brendanlong/sequential-transformer-lens-experiment

Checkpoints for Training a Transformer to Compose One Step Per Layer (and Proving It) Final checkpoints behind the writeup Training a Transformer to Compose One Step Per Layer (and Proving It). Code, analyses and the full experiment log: github.com/brendanlong/sequential-transformer-lens-experiment. Training curves: public wandb project. Every checkpoint is a torch.save dict {"step", "model_state_dict", "model_config"} loadable with torch.load(..., weights_only=True); each has a… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/sequential-transformer-lens-experiment.