asingh15/rubric-write-judge-distillation-partial

Rubric writer–judge distillation (partial)

This is a verified partial publication of the Amazon C11 adaptive-oracle distillation collection. It contains the first fully completed and materialized 100-reviewer train block. Both panel variants are included:

• —latent-state: 98 published panels; 2 filtered for insufficient gold-reward spread.
• —non-diverse: 91 published panels; 9 filtered for insufficient gold-reward spread.

The source is derived from the open-source Amazon Reviews 2023 corpus. The partial release has 189 published reviewer-panel trajectories and uses the qwen3_final_response loss-mask contract.

Configurations

from datasets import load_dataset

balanced = load_dataset("asingh15/rubric-write-judge-distillation-partial", split="train")
writer = load_dataset("asingh15/rubric-write-judge-distillation-partial", "writer", split="train")
judge = load_dataset("asingh15/rubric-write-judge-distillation-partial", "judge", split="train")

Row format

Each row contains:

• —stable example_id, pseudonymous reviewer_id, target_id, and trajectory_id;
• —role (rubric_writer or criterion_judge) and panel variant;
• —structured messages with content, reasoning_content, and step_loss_mask;
• —turn/criterion/candidate indices and a JSON metadata payload;
• —loss_mask_type = qwen3_final_response.

Context messages have step_loss_mask=0; the supervised assistant target has step_loss_mask=1. For an oracle-selected writer edit that did not originally submit, <submit/> is appended to that same assistant response without creating a synthetic extra turn.

Selection and filtering

The teacher writes structured rubrics one edit at a time. Candidate reward functions are evaluated against hidden oracle rewards, and the earliest maximum defined Spearman snapshot is selected. Panels with gold-reward spread at most 0.10, or with no rankable candidate reward function, are filtered rather than published. This partial contains 11 filtered panels, all due to insufficient gold-reward spread.

Partial-release scope

This repository currently contains block 0 only and therefore only a train split. It is intentionally marked partial while the full collection continues. Exact source manifests and artifact checksums are included under metadata/.