THU-KEG/SAEVerbalizer-Data

SAEVerbalizer Data

Evaluation data for [SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization](https://arxiv.org/abs/2608.13538). This release contains the paper's three evaluation sets for both the 27B verbalizer and the 1B-to-27B adapter-based verbalization system. Training data may be added in a later release.

Inference and Reference Agreement evaluation code is available in `THU-KEG/SAEVerbalizer`.

Data collections

The adapter maps the 1B SAE decoder directions into the hidden space used by the released `SAEVerbalizer-27B`. Feature IDs are local to the corresponding SAE and must not be exchanged between the two collections.

Evaluation sets

Each collection provides the same three paper-defined splits:

GTS contains globally sampled features satisfying the training qualification standard. LIG and GG satisfy the stricter gold qualification standard; LIG is sampled from the low-index region, while GG is sampled globally. Within each collection, the three sets are mutually disjoint.

Each file is a JSON list whose records contain:

• —id: the feature index in the collection's source SAE.
• —concept: the reference natural-language feature explanation from Neuronpedia.

Download

Download the 27B evaluation data:

hf download THU-KEG/SAEVerbalizer-Data \
  --repo-type dataset \
  --include "evaluation/*.json" \
  --local-dir data

Download the adapter evaluation data:

hf download THU-KEG/SAEVerbalizer-Data \
  --repo-type dataset \
  --include "adapter/1b_to_27b/evaluation/*.json" \
  --local-dir data

Source SAEs

• —Gemma Scope 2 27B, layer 16, width 262k, `l0_medium`
• —Gemma Scope 2 1B, layer 7, width 262k, `l0_medium`

Citation

@article{meng2026saeverbalizer,
  title   = {SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization},
  author  = {Meng, Weihan and Guo, Hongzhu and Jing, Yi and Liu, Dewen and Yao, Zijun and Wang, Xiaozhi and Hou, Lei and Li, Juanzi},
  journal = {arXiv preprint arXiv:2608.13538},
  year    = {2026}
}