Prisma-Multimodal/sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-1-hook_mlp_out-l1-0.0001

CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:0.0001

Training Details

• —Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
• —Layer: 1
• —Component: hookmlpout

Model Architecture

• —Input Dimension: 768
• —SAE Dimension: 49,152
• —Expansion Factor: x64 (vanilla architecture)
• —Activation Function: ReLU
• —Initialization: encodertransposedecoder
• —Context Size: 50 tokens

Performance Metrics

• —L1 Coefficient: 0.0001
• —L0 Sparsity: 249.8799
• —Explained Variance: 0.8170 (81.70%)

Training Configuration

• —Learning Rate: 0.0004
• —LR Scheduler: Cosine Annealing with Warmup (200 steps)
• —Epochs: 10
• —Gradient Clipping: 1.0
• —Device: NVIDIA Quadro RTX 8000

Experiment Tracking:

• —Weights & Biases Run ID: ob776mv6
• —Full experiment details: https://wandb.ai/perceptual-alignment/clip/runs/ob776mv6/overview
• —Git Commit: e22dd02726b74a054a779a4805b96059d83244aa

Citation

@misc{2024josephsparseautoencoders,
    title={Sparse Autoencoders for CLIP-ViT-B-32},
    author={Joseph, Sonia},
    year={2024},
    publisher={Prisma-Multimodal},
    url={https://huggingface.co/Prisma-Multimodal},
    note={Layer 1, hook_mlp_out, Run ID: ob776mv6}
}