miria0/EduFeedback

EduFeedback

<figure align="center"> <img src="alternate.png" alt="Alternating Population Strategy: both chosen and rejected are extracted from the same real conversation, with no synthetic LLM generation needed for either side" width="780"/> <figcaption><em>Alternating dataset example: a single curated multi-turn conversation yields a complete <code>(prompt, chosen, rejected)</code> triplet on its own &mdash; the direct early response becomes <code>chosen</code> and a later, less-direct response becomes <code>rejected</code>. Both sides come from the same real dialog, so no synthetic LLM generation is needed to fill in the <code>rejected</code> side.</em></figcaption> </figure>

EduFeedback is a synthetically generated, multi-turn conversational preference dataset in an educational tutoring setting, introduced in the COALA paper (Feng & Pilanci, ICML 2026). It is released together with the Alternating Population Strategy, a novel method for extracting high-quality (prompt, chosen, rejected) preference training triplets directly from multi-turn conversations — no external reward model, no extra generation calls, no PairRM-style scoring step.

Conversations are produced by GPT-4o acting as two agents — a tutor (agent1) and a student (agent2) — across eleven topics in science and philosophy, with varying mood/personality prompts and sampling temperatures (t = 0.2 – 0.9). Each conversation has 4–8 alternating utterances.

The dataset is released in three configurations: the raw conversations, and two preference-pair views derived from them.

TL;DR — three configs

No train/test split is provided; downstream users should generate their own splits to fit their setup.

Why the Alternating Population Strategy?

Curating real-world human conversations across domains is expensive, yet LLMs remain imperfect surrogates for genuine human responses. The standard preference-data convention only half-escapes this problem: it uses the costly curated dialog for the chosen side, but then falls back on an LLM to generate the rejected side &mdash; partially reintroducing the very synthetic surrogate the curation was meant to avoid. The Alternating Population Strategy sidesteps this: it populates both chosen and rejected directly from the same single real conversation, letting one curated dialog corpus do all the work and removing reliance on synthetic LLM-generated counterfactuals.

Concretely, the standard recipe for building a preference dataset from a corpus of conversations is to extract the prompt as all-but-the-final- turn, take the real final assistant turn as the chosen answer, then sample from a separate SFT baseline model around 5 times, and pass those samples to a third-party preference scorer such as PairRM to obtain a rejected.

This is:

• —slow and expensive — it requires hundreds of thousands of extra generations from an SFT model;
• —fragile — results are sensitive to both the SFT model and the choice of scoring model (PairRM, GPT-4-as-judge, learned reward model, ...);
• —wasteful — for a dataset of N multi-turn conversations, the convention extracts only N preference triplets (one per conversation), even though every assistant turn is in principle a usable training signal.

The Alternating Population Strategy (illustrated above) instead exploits a structural property of well-curated multi-turn educational dialogs:

In objective, domain-specific tutoring conversations, the early assistant turns answer the user's question directly, while later turns frequently drift into related follow-ups or tangents.

Given that asymmetry, you can synthesize a preference triplet from a single conversation without any external model:

• —prompt ← the user (agent2) question (plus any prior turns).
• —chosen ← the immediate next assistant (agent1) response — the most direct answer to that prompt.
• —rejected ← a later assistant response from the same conversation — still topically relevant and well-formed, but a less direct answer to the original question.

By sliding this construction across every assistant turn, a conversation of 3–4 turns yields 2–3 training triplets instead of just 1. Applied to the 26,621 EduFeedback conversations, this produces 65,606 preference training samples — roughly the same order of magnitude as UltraFeedback-Binarized (60,917), but at zero additional generation or scoring cost.

Why this matters

• —Maximum value from curated dialog data. Real human conversations are expensive to collect; the alternating method extracts every usable preference pair from each dialog instead of throwing most of it away.
• —No synthetic `rejected` responses. Both sides of the pair come from the same real conversation. The strategy generalizes naturally to costly human-curated conversation corpora, where avoiding an LLM-generated rejected side is most valuable.
• —No reward model required. Preference signal comes from conversation structure rather than from a learned or third-party ranker, removing a major source of pipeline brittleness and bias.
• —No extra inference calls. All triplets are extracted from the existing dialog; no SFT-baseline sampling pass is needed.
• —Higher data efficiency. "Since data for machine learning is becoming increasingly scarce, it is critical to extract the maximum amount of value possible from available datasets" (COALA paper, App. E.2). The alternating method recovers ~2.5× more triplets per conversation than the standard one-per-conversation convention.
• —Validated by COALA training. COALA fine-tuned on this dataset shows stable, monotonically increasing reward margins during training (Figure 3 in the paper), in contrast to the noisier DPO trajectories on the same data.
• —Validated by humans. In the paper's 107-participant double-blind human study, COALA fine-tuned on EduFeedback-Alternate achieved a 39.1% real-human win rate on the EduFeedback task, the highest of any compared method (ORPO 15.5%, DPO 28.8%, SFT 16.6%) — direct evidence that the heuristic captures genuine preference signal rather than just surface statistics.

Worked example

(Reproduced from Appendix E.3 of the COALA paper.)

Original multi-turn conversation

System: You are Dolphin, a helpful AI tutor. User (agent2): Hi! I'm studying for my music class and need some help understanding the different periods in classical music. Can you explain the main characteristics that differentiate the Baroque period from the Romantic period? Assistant (agent1, response i): Certainly! The Baroque period (roughly 1600–1750) is characterized by intricate compositions, ornate embellishments, and a strong emphasis on harmony and counterpoint. ... In contrast, the Romantic period (roughly 1800–1910) focuses more on emotional expression and individuality. ... User (agent2): I need more help to study for my music class tomorrow. What if a question on the quiz asks me to enumerate the main differences between the Baroque and Romantic periods? Assistant (agent1, response i+2): During the Classical period (roughly 1750–1820), the role of the conductor was often filled by a leading musician within the ensemble, such as the first violinist or the keyboard player ...

Extracted preference triplet

• —Prompt: the user's first question about Baroque vs Romantic periods.
• —Chosen: response i — directly answers the Baroque vs Romantic question.
• —Rejected: response i+2 — still informative and topically related, but drifts into the role of the conductor in the Classical period instead of the requested comparison.

Configurations — schemas

conversations

pairs-single

pairs-alternating

This format is directly consumable by HuggingFace TRL preference trainers (DPO, ORPO, SimPO, etc.).

Usage

from datasets import load_dataset

# Raw conversations
conv = load_dataset("miria0/EduFeedback", "conversations", split="train")

# Single-pair-per-conversation preference data
single = load_dataset("miria0/EduFeedback", "pairs-single", split="train")

# Full alternating-population preference data (used in COALA)
alt = load_dataset("miria0/EduFeedback", "pairs-alternating", split="train")

Generation details

• —Generator: GPT-4o (OpenAI), accessed via the OpenAI API.
• —Sampling temperature: uniformly drawn from {0.2, …, 0.9} per conversation.
• —Conversation length: 4–8 alternating utterances between agent1 (tutor) and agent2 (student).
• —Eleven topics span science and philosophy domains.
• —Student mood is randomly varied per conversation to simulate realistic human conversational behavior.

Further details on the dataset construction and the Alternating Population Strategy are provided in Appendix E.3 of the COALA paper.

Intended use

EduFeedback is intended as a benchmark and training corpus for preference fine-tuning of language models in objective, knowledge-grounded domains (correctness, helpfulness, safety) rather than for subjective style transfer. It is well-suited for evaluating reference-free / single-GPU preference optimization methods such as COALA, DPO, ORPO, and SimPO.

Limitations and disclaimers

• —The data is fully synthetic and generated by GPT-4o. It may contain factual errors, stylistic artifacts, or biases inherited from the generator. It should not be relied upon as a source of factual information.
• —The dataset was produced using OpenAI models. Users must comply with the applicable OpenAI usage policies and terms of service when training or evaluating models on this data.
• —The rejected responses in the preference configs are not incorrect answers — they are simply less direct answers to the original prompt. This matches the intent of the Alternating Population Strategy and is what makes the preference signal subtle and content-grounded (rather than rewarding obvious low-quality contrasts), but it should be kept in mind for downstream analyses.

License

Apache-2.0.

Citation

If you use this dataset, please cite the COALA paper:

@inproceedings{feng2026coala,
  title     = {Convex Optimization for Alignment and Preference Learning on a Single GPU},
  author    = {Feng, Miria and Pilanci, Mert},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
}