Prisma-Multimodal/celeba-sae-top_k-64-patches_only-layer_1-hook_resid_post-64-86

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• —Layer: 1
• —Layer Type: hookresidpost
• —Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• —Dictionary Size: 49152
• —Input Dimension: 768
• —Expansion Factor: 64
• —CLS Token Only: False

Training

• —Training Images: 324085
• —Learning Rate: 0.0006
• —L1 Coefficient: 0.0002
• —Batch Size: 4096
• —Context Size: 49

Performance Metrics

Sparsity

• —L0 (Active Features): 64.0000
• —Dead Features: 0
• —Mean Log10 Feature Sparsity: -5.1483
• —Features Below 1e-5: 38050.0000
• —Features Below 1e-6: 267.0000
• —Mean Passes Since Fired: 61.2594

Reconstruction

• —Explained Variance: 0.8683
• —Explained Variance Std: 0.0462
• —MSE Loss: 0.0014
• —L1 Loss: 0
• —Overall Loss: 0.0014

Training Details

• —Training Duration: 1157 seconds
• —Final Learning Rate: 0.0000
• —Warm Up Steps: 500
• —Gradient Clipping: 1

Additional Information

• —Original Checkpoint Path: /network/scratch/p/praneet.suresh/celebacheckpoints/14fc7e55-tinyclipsae16hyperparamsweeplr/nimages324169.pt
• —Wandb Run: https://wandb.ai/perceptual-alignment/celeba-sweep-topk-patchesalllayers/runs/wul48i6z
• —Random Seed: 42