CoolFace
Datasetpublic

aladinDJ/tulu-3-sft-mix-annotated-old

MagPie-Annotated Tülu-SFT-Mix A MagPie-annotated version of the Tülu-SFT-Mix dataset, with fine-grained tags for task category, conversation depth, instruction quality, response reward, safety, language, and difficulty, enabling in-depth analyses and facilitating downstream mixture design. 🚀 Dataset Overview We take the original Tülu-3 SFT Mix and enrich every example using the MagPie annotation pipeline (judge model: Llama-3.3-70B-Instruct). Each sample now… See the full description on the dataset page: https://huggingface.co/datasets/aladinDJ/tulu-3-sft-mix-annotated-old.

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes101downloads
Dataset Card

MagPie-Annotated Tülu-SFT-Mix

A MagPie-annotated version of the Tülu-SFT-Mix dataset, with fine-grained tags for task category, conversation depth, instruction quality, response reward, safety, language, and difficulty, enabling in-depth analyses and facilitating downstream mixture design.

🚀 Dataset Overview

We take the original Tülu-3 SFT Mix and enrich every example using the MagPie annotation pipeline (judge model: Llama-3.3-70B-Instruct). Each sample now carries:

  • —Task Category (12 classes: Coding & Debugging, Reasoning, Information Seeking, …)
  • —Conversation Depth (single-turn vs. multi-turn)
  • —Input Quality (very poor ↔ excellent)
  • —Response Quality (Instruct Reward)
  • —Safety Label (via Llama-Guard 2)
  • —Language (e.g. English, Chinese, …)
  • —Difficulty (very easy ↔ very hard)
  • —Source Attribution & Sample ID (retained from the original)

These rich annotations let you quantify instruction fidelity, response adequacy, and safety, illuminating where Tülu and other mixes overlap, diverge, and complement each other.

🔍 Annotation Pipeline

  1. 1.Unified Tagging: All examples from Tülu-3-SFT-Mix were fed to MagPie prompts with in-context templates and an error-tolerant JSON parser to ensure consistent outputs.
  2. 2.Template Enforcement: In-prompt examples + tolerant parser reduced normalization errors by 5–15%. Manual corrections handled the remainder.
  3. 3.Multi-Turn Support: Prompts were adapted to ingest full conversation histories using Llama-3.3-70B’s 128k context window—preventing truncation and ensuring faithful tagging.
  4. 4.Safety Assessment: Each response is scored via Llama-Guard 2 to flag unsafe content.
Coverage: ≥ 97 % of the original samples were successfully tagged (annotation failure < 3 %).

📖 Citation

Will be provided soon. <!-- If you use this dataset, please cite:

bibtex
@inproceedings{your2025magpie,
  title     = {MagPie-Annotated Tülu-SFT-Mix: Fine-Grained Annotation of Instruction-Tuning Data},
  author    = {Your Name et al.},
  booktitle = {NeurIPS 2025},
} -->