Prisma-Multimodal/imagenet-sae-top_k-64-patches_only-layer_10-hook_resid_post-64-77

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• —Layer: 10
• —Layer Type: hookresidpost
• —Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• —Dictionary Size: 49152
• —Input Dimension: 768
• —Expansion Factor: 64
• —CLS Token Only: False

Training

• —Training Images: 1299936
• —Learning Rate: 0.0001
• —L1 Coefficient: 0.0002
• —Batch Size: 4096
• —Context Size: 49

Performance Metrics

Sparsity

• —L0 (Active Features): 64.0000
• —Dead Features: 0
• —Mean Log10 Feature Sparsity: -3.1090
• —Features Below 1e-5: 9
• —Features Below 1e-6: 0
• —Mean Passes Since Fired: 0.2095

Reconstruction

• —Explained Variance: 0.7779
• —Explained Variance Std: 0.0779
• —MSE Loss: 0.0056
• —L1 Loss: 0
• —Overall Loss: 0.0056

Training Details

• —Training Duration: 3927 seconds
• —Final Learning Rate: 0.0000
• —Warm Up Steps: 500
• —Gradient Clipping: 1

Additional Information

• —Original Checkpoint Path: /network/scratch/p/praneet.suresh/celebacheckpoints/df079963-tinyclipsae16hyperparamsweeplr/nimages1300020.pt
• —Wandb Run: https://wandb.ai/perceptual-alignment/imagenet-sweep-topk-patchesalllayers/runs/swu2oebm
• —Random Seed: 42