Prisma-Multimodal/imagenet-sae-top_k-64-patches_only-layer_7-hook_resid_post-128-79

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• —Layer: 7
• —Layer Type: hookresidpost
• —Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• —Dictionary Size: 49152
• —Input Dimension: 768
• —Expansion Factor: 64
• —CLS Token Only: False

Training

• —Training Images: 1299936
• —Learning Rate: 0.0001
• —L1 Coefficient: 0.0002
• —Batch Size: 4096
• —Context Size: 49

Performance Metrics

Sparsity

• —L0 (Active Features): 128.0000
• —Dead Features: 0
• —Mean Log10 Feature Sparsity: -2.8098
• —Features Below 1e-5: 0
• —Features Below 1e-6: 0
• —Mean Passes Since Fired: 0.0573

Reconstruction

• —Explained Variance: 0.7958
• —Explained Variance Std: 0.0439
• —MSE Loss: 0.0024
• —L1 Loss: 0
• —Overall Loss: 0.0024

Training Details

• —Training Duration: 4202 seconds
• —Final Learning Rate: 0.0000
• —Warm Up Steps: 500
• —Gradient Clipping: 1

Additional Information

• —Original Checkpoint Path: /network/scratch/p/praneet.suresh/celebacheckpoints/165d9ce1-tinyclipsae16hyperparamsweeplr/nimages1300020.pt
• —Wandb Run: https://wandb.ai/perceptual-alignment/imagenet-sweep-topk-patchesalllayers/runs/zdtnf9b0
• —Random Seed: 42