Datasetpublic

syvb/nla-qwen2.5-7b-L20-matryoshka-warmstart-sonnet46

NLA Qwen2.5-7B L20 warm-start data — Sonnet-4.6 "matryoshka" explanations + activations Re-warm-start dataset for the Natural Language Autoencoders Qwen2.5-7B (layer-20) AV/AR pair. Pairs Qwen2.5-7B-Instruct layer-20 residual-stream activations with the Claude Sonnet-4.6 explanations from ceselder/nla-matryoshka-warmstart-sonnet46. The source dataset is text-only (explanations keyed by custom_id, no vectors). This dataset adds the missing activations: for each av-*/ar-* row the… See the full description on the dataset page: https://huggingface.co/datasets/syvb/nla-qwen2.5-7b-L20-matryoshka-warmstart-sonnet46.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes25downloads

Dataset Card

NLA Qwen2.5-7B L20 warm-start data — Sonnet-4.6 "matryoshka" explanations + activations

Re-warm-start dataset for the Natural Language Autoencoders Qwen2.5-7B (layer-20) AV/AR pair. Pairs Qwen2.5-7B-Instruct layer-20 residual-stream activations with the Claude Sonnet-4.6 explanations from `ceselder/nla-matryoshka-warmstart-sonnet46`.

The source dataset is text-only (explanations keyed by custom_id, no vectors). This dataset adds the missing activations: for each av-*/ar-* row the activation is Qwen2.5-7B-Instruct's layer-20 hidden state at the last token of `input_text`. input_text is exactly token_ids[:position] — verified: it retokenizes to the custom_id position count for 100% of rows — so this is reproducible from the text alone, no source corpus needed (the original Ultra-FineWeb slice is no longer hosted / has drifted).

Files

file	rows	use
`av_sft.parquet`	219992	actor (AV) warm-start SFT — train
`av_eval.parquet`	5007	actor eval holdout (document-level)
`ar_sft.parquet`	219600	critic (AR) warm-start SL — train
`ar_eval.parquet`	5008	critic eval holdout (document-level)
`base_av.parquet`	224999	master: vectors+explanation+text+custom_id (all av)
`base_ar.parquet`	224608	master: vectors+explanation+text+custom_id (all ar)

Each parquet ships a .nla_meta.yaml sidecar. SFT/eval parquets are stage-3 format (drop-in for configs/actor_sft.sh / critic_sft.sh). The base_* masters carry the raw activation vectors (norm="none") + api_explanation + detokenized_text_truncated

—custom_id; run nla.datagen.stage3_build on them to regenerate SFT parquets with other templates, or to re-split.

Holdout

~10k samples held out for eval (≈5k av + ≈5k ar), split at the document level (whole docs, seed 42) so no training position shares a document with an eval position.

Provenance

—base model Qwen/Qwen2.5-7B-Instruct, layer 20, d_model 3584, raw vectors (norm="none")
—explanations claude-sonnet-4-6 (av-/ar- succeeded rows of the matryoshka dataset)