tintin1027/atomic-metrics-six-task-preferences

Six-task benchmark inputs

Seed 17. No demographic conditioning. Each task has shared train100.jsonl and test500.jsonl for Atomic Metrics, five judge variants, and learned baselines. Pair plans cover all 100 training rows once. Atomic Metrics extraction and BT/LR fitting use train100. Judges use the same test500. RM and WIMHF in the matched-data comparison use train100; rmtrainfull is an explicitly separate expanded-data setting and must not be described as train100. WIMHF-adapted copies are in wimhf/ with label 1 for A, 0 for B. RM consumes prompt/chosen/rejected; judge/method inputs additionally include responsea/responseb/human_preference.

Human preference labels are preserved. IDs are task-prefixed content hashes; sourcesampleid retains provenance. Opposing labels on identical unordered pairs are excluded. Prompt/response connected components (plus upstream SI split groups) never cross train/test within a task. DI has fewer than 500 unique test prompts, so its test includes multiple distinct response pairs per prompt. Training prompt selection prefers the previous train100 where eligible. Existing source files and running demographic experiments are untouched.

Do not tune on test500. Use training-only cross-validation for hyperparameters. These are per-task splits; do not pool tasks for training without an additional cross-task overlap audit. No new model/API runs are started by preparation.

Split sizes

Loading

from datasets import load_dataset
ds = load_dataset("tintin1027/atomic-metrics-six-task-preferences", "creative_writing")

train_full includes the main 100-row training subset; it is not an additional disjoint split. All models/methods share the same 500-row test set within each task. These data contain full prompts and responses; experiment-time truncation is configured separately.

Sources and terms

This is a derived research dataset with mixed upstream terms, not a newly licensed replacement for those sources. Preserve and follow each original dataset's terms:

• —Stanford Human Preferences: https://huggingface.co/datasets/stanfordnlp/SHP (Reddit-derived preferences; consult original terms).
• —OpenAssistant OASST1 and OASST2: https://huggingface.co/datasets/OpenAssistant/oasst1 and https://huggingface.co/datasets/OpenAssistant/oasst2 (Apache-2.0).
• —Community Alignment and Writing Preference Bench: original source names are retained in source_dataset; their original terms continue to apply.
• —Research Ideation Arena: https://huggingface.co/datasets/yolo1213811/research-ideation-arena-dataset and https://github.com/foss12138/Research-Ideation-Arena . The source card specifies CC BY-NC 4.0 while its bundled license specifies CC BY 4.0; conservatively treat SI as noncommercial pending clarification. The original bundled license is included in licenses/.

Preference annotations come from upstream human judgments/votes, not HY3. HY3 supplied NCW/CW category labels only. Writing classification was stopped before all prompts were classified; these splits therefore reflect only the completed subset, not all available writing data. Rows flagged outside writing were excluded. Model-assigned task categories can be wrong. Source and task distributions are imbalanced; chosen responses are not guaranteed correct or safe.

The scientific-ideation expanded training pool excludes 607 rows sharing held-out components, retaining 2,440. Daily Ideation test has 500 distinct pairs but only 281 unique prompts. Use training-only validation, and account for prompt clustering in statistical analysis. Existing demographic experiments are not part of this release.