seoirsem/CHUNKY-tulu3-SFT-25k-attributes

SURF Attributes

Minimal dataset for running SURF (Surfacing Unintended Response Failures).

Paper: Chunky Post-Training

Usage

uv run -m surf.cli.main sweep \
    --attributes seoirsem/CHUNKY-tulu3-SFT-25k-attributes \
    --rubric rubrics/rebuttal.yaml \
    -o results/

Fields

• —prompt: The query text
• —sae_attributes: List of semantic attribute cluster summaries

How it works

Each prompt was analyzed to extract 10 raw attributes describing its content, style, and formatting. These attributes were embedded and clustered into ~25k semantic categories. Each category was summarized into a human-readable description. The sae_attributes field contains the cluster summaries relevant to each prompt, which the EM loop samples from during query generation.

Full Dataset

For research and extension, see seoirsem/CHUNKY-tulu3-SFT-25k-attributes-full which includes:

• —All original fields (response, raw attributes, etc.)
• —Cluster centroids for embedding-based lookup
• —Cluster summaries and statistics

Attribution

This dataset is derived from Tülu 3 SFT Mixture by Allen AI, licensed under ODC-BY.

Modifications: Prompts were processed to extract semantic attributes, which were embedded, clustered into ~25k categories, and summarized.

Please cite the original Tülu 3 paper:

Lambert et al. (2024). "Tülu 3: Pushing Frontiers in Open Language Model Post-Training"