Prisma-Multimodal/imagenet-sae-top_k-64-patches_only-layer_7-hook_resid_post-256-80

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• —Layer: 7
• —Layer Type: hookresidpost
• —Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• —Dictionary Size: 49152
• —Input Dimension: 768
• —Expansion Factor: 64
• —CLS Token Only: False

Training

• —Training Images: 1299936
• —Learning Rate: 0.0000
• —L1 Coefficient: 0.0002
• —Batch Size: 4096
• —Context Size: 49

Performance Metrics

Sparsity

• —L0 (Active Features): 256.0000
• —Dead Features: 0
• —Mean Log10 Feature Sparsity: -2.7591
• —Features Below 1e-5: 0
• —Features Below 1e-6: 0
• —Mean Passes Since Fired: 0.0877

Reconstruction

• —Explained Variance: 0.8019
• —Explained Variance Std: 0.0403
• —MSE Loss: 0.0023
• —L1 Loss: 0
• —Overall Loss: 0.0023

Training Details

• —Training Duration: 4266 seconds
• —Final Learning Rate: 0.0000
• —Warm Up Steps: 500
• —Gradient Clipping: 1

Additional Information

• —Original Checkpoint Path: /network/scratch/p/praneet.suresh/celebacheckpoints/9a718734-tinyclipsae16hyperparamsweeplr/nimages1300020.pt
• —Wandb Run: https://wandb.ai/perceptual-alignment/imagenet-sweep-topk-patchesalllayers/runs/seflxmma
• —Random Seed: 42