tintin1027/research-ideation-arena-si-rm

Research Ideation Arena — Scientific Ideation RM Splits

Derived from Research Ideation Arena, revision f5704385bd66781d504e44810a9a8b56c1623b7a. Original authors: Zhiyu Chen et al. See the paper and official code.

Splits and evaluation caveat

Train: 3,047 preference pairs. Test: 500 fixed preference pairs. All remaining pairs from the 3,547-pair filtered pool are assigned to training. Exact sample/pair overlap is zero, but 607 training rows share a connected literature-context/response group with the test split. This default split is therefore not a leakage-free held-out evaluation. Filter training with shares_test_group == False for the 2,440-row group-disjoint training candidate pool, reserving validation data from its training groups separately.

from datasets import load_dataset

ds = load_dataset("tintin1027/research-ideation-arena-si-rm")
train = ds["train"]
test = ds["test"]
train_group_disjoint = train.filter(lambda row: not row["shares_test_group"])

Human labels and processing

The original 6,441 records contain expert pairwise judgments. We use D0 (Overall Quality) only: 5,757 A/B votes, excluding 540 Tie and 144 BothBad records. Duplicate response pairs under normalized, order-independent literature contexts are collapsed; 515 pairs with conflicting decisive human votes are excluded. No LLM-generated labels are added. The fixed test set was selected using seed 17, one pair per connected component of shared contexts or identical response texts. A/B positions in that test set were randomized. All eligible non-test pairs are now included in training. This is an unambiguous binary subset, not the original benchmark's full-vote or soft-accuracy protocol.

prompt contains a fixed research-proposal instruction followed by the original literature context. Responses and contexts are not truncated. Core fields: sample_id, source_dataset, category, task, split, prompt, chosen, rejected, response_a, response_b, human_preference. Provenance: source_record_indices, source_record_index, source_dimension_results, source_human_preference, context_id, split_group_id, position_swapped, generator names. source_dimension_results always retains original A/B positions. Generator-name fields follow the exported response positions. shares_test_group flags training records associated with a test group; it is false for the test rows themselves. Do not feed preference labels, chosen/rejected designations, or provenance into the judge prompt.

License and attribution

The upstream dataset card labels the data CC BY-NC 4.0, while its bundled LICENSE says CC BY 4.0. This release conservatively retains the dataset card's noncommercial designation and includes the original license file verbatim as UPSTREAM_LICENSE.txt. Consult the upstream authors to resolve the discrepancy before commercial use. The transformations and split policy above are modifications by this derivative release; retain attribution to the original authors.