CoolFace
Datasetpublic

achand45/gemma-3-12b-it-nla-data

Gemma-3-12B-IT NLA training data — blocks 24 / 32 / 40 / 47 Training data for the achand45/gemma-3-12b-it-nla-L* natural language autoencoders: residual-stream activations from google/gemma-3-12b-it paired with the prompts and gold explanations used to train the verbalizer (AV) and reconstructor (AR). One directory per layer. The four arms are the same rows in the same order — only activation_vector and activation_layer differ — so they are directly comparable. config rows… See the full description on the dataset page: https://huggingface.co/datasets/achand45/gemma-3-12b-it-nla-data.

sourceHugging Facegemmaupdated 1mo agoView on Hugging Face
0likes15downloads
Dataset Card

Gemma-3-12B-IT NLA training data — blocks 24 / 32 / 40 / 47

Training data for the achand45/gemma-3-12b-it-nla-L* natural language autoencoders: residual-stream activations from google/gemma-3-12b-it paired with the prompts and gold explanations used to train the verbalizer (AV) and reconstructor (AR).

One directory per layer. The four arms are the same rows in the same order — only activation_vector and activation_layer differ — so they are directly comparable.

configrowsfilesize
L*_ar_sft247,358L*/ar_sft_shuf.parquet~1.6 GB
L*_av_sft247,261L*/av_sft_shuf.parquet~1.6 GB
L*_rl499,846L*/rl_shuf.parquet~3.2 GB

Schema

columntypemeaning
activation_vectorfloat32[3840]the residual-stream activation — the thing being encoded
activation_layerintdecoder block it was read from (24, 32, 40 or 47 of 48)
detokenized_text_truncatedstrthe input prefix, truncated to end exactly at the extraction token; run it through the base model to recover the activation
n_raw_tokensint1-indexed token count of the prefix; the activation is at position n_raw_tokens - 1
doc_idstrprovenance, <corpus>:<split>:<doc_index>
promptstrtraining text — AV/RL: the verbalizer prompt carrying the injection marker; AR: the explanation mapped back to the vector
responsestrav_sft only — the gold explanation

Each parquet ships a .nla_meta.yaml sidecar recording the extraction settings, prompt templates, injection token ids and parent dataset.

Splits

The val split is a doc-disjoint holdout — 204 documents, every row of them excluded from training. A row-level split leaks badly here: the corpus is row-shuffled and each document contributes ~10 rows.

full is the whole file, val rows included. Reconstruct the exact training split by removing the val doc_ids:

python
from datasets import load_dataset

d = load_dataset("achand45/gemma-3-12b-it-nla-data", "L32_av_sft")
val_docs = set(d["val"]["doc_id"])
train = d["full"].filter(lambda r: r["doc_id"] not in val_docs)

RL used no val file; its held-out eval is the tail of rl_shuf.parquet (eval_skip_rows: 449846, i.e. the last 50,000 rows).

Provenance

Text, ordering and gold explanations come from `ceselder/qwen3-8b-nla-L24-finefineweb-100k` (FineFineWeb, 100k docs, ~10 positions each). Only activation_vector and activation_layer are ours: activations were re-harvested from google/gemma-3-12b-it at each layer with EasyNLA's nla.datagen. No explanations were regenerated, so the arms share identical supervision text.

Models trained on this

`-nla-L32` · `-nla-L40` · `-nla-L47`. The L24 arm is still training.