enjalot/sae-all-MiniLM-L6-v2-FineWeb-RedPajama-Pile-150M

SAE for all-MiniLM-L6-v2 (FineWeb + RedPajama + Pile, 150M)

Sparse Autoencoder trained on sentence embeddings from all-MiniLM-L6-v2, decomposing 384-dimensional dense embeddings into sparse, interpretable features.

Available Models

Recommended: `128_4` — only 402 active features but best accuracy on hard tasks (CLINC150 79.6%, BANKING77 86.5%), most distinct features (MMCS 0.193), and half the parameters. Best balance of quality and efficiency.

Quick Start

from latentsae import Sae
from sentence_transformers import SentenceTransformer

# Load SAE
sae = Sae.load_from_hub("enjalot/sae-all-MiniLM-L6-v2-FineWeb-RedPajama-Pile-150M", "64_8")

# Embed text
emb_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = emb_model.encode(["Your text here"], normalize_embeddings=True)

# Extract sparse features
import torch
features = sae.encode(torch.tensor(embeddings))
print(f"Top feature indices: {features.top_indices}")
print(f"Top feature activations: {features.top_acts}")

Training Details

• —Embedding model: sentence-transformers/all-MiniLM-L6-v2 (384D)
• —Training data: 150M embeddings (50M each):
• —FineWeb-edu 10BT sample (120-token chunks)
• —RedPajama-Data-V2 10B sample (120-token chunks)
• —Pile uncopyrighted (120-token chunks)
• —Architecture: TopK SAE, k=64, 8x expansion (3,072 features), 2.4M parameters
• —Training: auxkalpha=1/32, deadfeature_threshold=50K, cosine LR schedule
• —Hardware: A10G on Modal, 54 minutes, ~$1

Evaluation (Probe Accuracy)

Linear probes on SAE sparse features vs raw embeddings:

Feature Quality

Part of the latent-* ecosystem

• —latent-sae — SAE training framework
• —latent-scope — Interactive dataset exploration
• —latent-taxonomy — SAE feature visualization