Prisma-Multimodal/imagenet-sae-top_k-64-patches_only-layer_4-hook_resid_post-64-79

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• —Layer: 4
• —Layer Type: hookresidpost
• —Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• —Dictionary Size: 49152
• —Input Dimension: 768
• —Expansion Factor: 64
• —CLS Token Only: False

Training

• —Training Images: 1299936
• —Learning Rate: 0.0042
• —L1 Coefficient: 0.0002
• —Batch Size: 4096
• —Context Size: 49

Performance Metrics

Sparsity

• —L0 (Active Features): 64.0000
• —Dead Features: 0
• —Mean Log10 Feature Sparsity: -4.0979
• —Features Below 1e-5: 5956
• —Features Below 1e-6: 271
• —Mean Passes Since Fired: 12.9838

Reconstruction

• —Explained Variance: 0.7914
• —Explained Variance Std: 0.0418
• —MSE Loss: 0.0017
• —L1 Loss: 0
• —Overall Loss: 0.0017

Training Details

• —Training Duration: 4171 seconds
• —Final Learning Rate: 0.0000
• —Warm Up Steps: 500
• —Gradient Clipping: 1

Additional Information

• —Original Checkpoint Path: /network/scratch/p/praneet.suresh/celebacheckpoints/abb1ffc0-tinyclipsae16hyperparamsweeplr/nimages1300020.pt
• —Wandb Run: https://wandb.ai/perceptual-alignment/imagenet-sweep-topk-patchesalllayers/runs/3dc5uf9y
• —Random Seed: 42