siddharthmb/llama1b-layer07-fineweb-128m-activations

llama1b-layer07-fineweb-128m-activations (PRIVATE archive)

Residual-stream activation store harvested from meta-llama/Llama-3.2-1B at layer 7, used as the training store for the GLP (Generative Latent Prior, arXiv:2602.06964) training-pipeline validation experiment (Silico thread experiment #46 / exp_01kxqnxk5yftyrf9cyt0a5gga8). Kept private out of Llama-license caution (derived from Llama weights' internal states).

Contents

• —chunks/chunk_XXXXXX/activations.safetensors — 1,099 chunks, ~500 MB each, bf16 [items, 2048] tensors (goodfire-core ActivationDataset v0.3.4 chunk format, mode=token).
• —metadata.json — authoritative chunk inventory (134,019,975 activations total).
• —index.sqlite — goodfire-core ActivationIndex (item → provenance mapping).
• —rep_statistics.pt — per-coordinate running statistics.
• —stats/ — harvest manifests + per-partition stats (from train128m_stats/).

Harvest protocol (paper protocol, validated in experiment #46)

• —Source data: HuggingFaceFW/fineweb sample-10BT, documents 30,000–268,000.
• —Per document: tokenize, truncate to 2048 tokens, capture all non-BOS positions at layer 7 of Llama-3.2-1B, bf16.
• —Harvested 2026-07-17 with goodfire-core gf-harvest (2 SLURM partitions; see stats/harvest_manifest_partition_{0,1}.json for exact configs/seeds).

Role

This store reproduced the GLP authors' released glp-llama1b-d3 training trajectory to within 0.3% val loss at all 8 released checkpoints (experiment #46's validation gate). It is archived here so the gate can be re-run without a ~10 GPU-hour re-harvest. Load with goodfire-core ActivationDataset after downloading the tree to disk.