Solshine/gemma-4-e2b-nla-av_sft-v0_1_x-short-hybrid-labels

## ⚠ CORRECTED 2026-05-16 — label format mismatch flagged This corpus's label format does NOT match Anthropic's NLA methodology. Median response length is 6 words with ≤7-word "explanation" tags (e.g., `"<explanation>

Galileo refuting Aristotle's gravity </explanation>"). Anthropic's NLA training data uses multi-paragraph ~80–120-word explanations with bolded topic headings (see https://huggingface.co/kitft/Llama-3.3-70B-NLA-L53-av` for an example of the expected label format).

This dataset was created as a deliberate test of a "short labels might help" hypothesis (H23 in the original v0.1.x trial series). H23 was originally reported as "refuted" — but the trial that refuted it also used an out-of-distribution injection_scale (see the parent v0.1.x trajectory release's CORRECTED block + FINDINGS.md §F72 in the source research repo). H23's refutation is now considered DOUBLY-WRONG — tested with both OOD injection AND non-Anthropic-format labels. Going forward we recommend training NLAs on long-label corpora (see Solshine/gemma-4-e2b-nla-av_sft-v0_1_x-gemini-persona-audit for the in-format alternative). This dataset stays published as a methodology-mismatch artifact and for reproducing the original H23 test. No data being taken down. Append-only.

license: cc-by-4.0 language:

• —en ---

av_sft — Hybrid Short-Label Dataset (Opus 4.6 / Sonnet 4.6 / Gemini 2.5 Pro / DeepSeek V4 Flash)

Location: data/stage3_v0_1_full_opus_short/av_sft.parquet Generator: experiments/v8_nla_local/stage2_opus_short_labels.py Label date: 2026-05-15 Rows: 4,734 (same schema as stage3_v0_1_full/av_sft.parquet + labeler_model column)

Purpose

Tests the label-format-as-lever hypothesis: the original Gemini labels are 200+ char paragraph-length elaborations (e.g., "The model tracks the 'unresolved' status of the 'unresolved' attribute, which..."). Short labels (≤5 words) may force the AV to use the activation signal rather than learn generic template surface area from the long-form labels.

Labeler model breakdown

Total re-labeled: 2,633 rows (55.6%) (+ ~2,101 deepseek in progress)

Label format

All three labelers were given the same prompt (the source text from stage0 chunks, truncated to 1,200 characters) and asked to produce a ≤5 word tag describing what concept the activation vector most likely encodes at the final token position.

Examples:

• —opus-4-6: "Galileo refuting Aristotle's gravity"
• —sonnet-4-6: "police body camera accountability"
• —gemini-2.5-pro: "Kidney transplant candidate eligibility"

The labeler_model column in the parquet records which model generated each label, enabling downstream A/B analysis of labeler quality.

Schema

Same as stage3_v0_1_full/av_sft.parquet with one additional column:

Known issues and fixes applied

Fixed

1. 1.Corrupted chunk row_001798.json — truncated extra -4-6"} tail appended to valid JSON. Repaired 2026-05-15.
2. 2.Missing `labeler_model` field on 862 Opus chunks — backfilled to opus-4-6 in both the chunk files and rebuilt parquet.

Unresolved

1. 1.261 Opus failures (opus_failed): 66 unique arxiv doc_ids, each with 4 consecutive failed rows. Likely Opus timing out on longer arxiv source texts.
2. 2.28 Gemini failures (gemini_failed): 7 kidney-donor docids at the tail of Gemini's run where it hit `QUOTAEXHAUSTED`.
3. 3.2 Sonnet failures (claude_failed): Negligible.

To fill the 291 gaps: ~36 minutes of Sonnet calls at 7.5s/row. The chunk directory is restart-safe — re-running the script skips completed rows.

Chunk directory

Raw per-row JSON files live in chunks/ (row_{index:06d}.json), each containing:

{
  "idx": 0,
  "doc_id": "fwe_00000053",
  "label": "Galileo refuting Aristotle's gravity",
  "labeler_model": "opus-4-6"
}

Chunks with empty labels have an "error" field instead.

Rebuilding the parquet

python experiments/v8_nla_local/stage2_opus_short_labels.py --n-rows 4734

The script reads chunks/ and rebuilds av_sft.parquet. It will also resume labeling any unfilled rows when run with no --n-rows limit.