forreview43/ai-vs-human-rubric-companion-data

Companion dataset for the AI-vs-human rubric study

This dataset is the data side of an anonymous NeurIPS submission. It pairs with a separate anonymous code repository that contains the runnable scripts, validators, and documentation. The two artifacts together reproduce every paper-facing headline number without re-running any API-backed stage. The code URL for review is https://anonymous.4open.science/r/codereviewer-47F3/README.md.

How to use

Download this dataset into the ./data/ directory of the companion code repository. The validation and pipeline scripts expect the layout below to live at the release-relative path data/.

export HF_DATASET_REPO=forreview43/ai-vs-human-rubric-companion-data
./code/bin/00_fetch_data            # convenience wrapper around huggingface-cli download
python3 code/validation/validate_release_metrics.py --section all

Layout

• —croissant.jsonld Croissant metadata file for the OpenReview Croissant upload field, including Responsible AI fields.
• —reviewer_sample/ representative sample for the OpenReview Dataset Large URL field.
• —SAMPLE_CREATION.md explains how the reviewer sample was selected.
• —HF_UPLOAD_CHECKLIST.md lists the exact OpenReview fields to fill after upload.
• —paper_release/ paper-facing inputs, embeddings, derived rubric and finding artifacts.
• —inputs/ benchmark cases, the 100-case sample, and the seed file.
• —embeddings/ text embedding cache used in matching and coverage steps.
• —rubrics/original/ the released MoReBench human rubric.
• —rubrics/ai_rubrics/ 13 model-generated rubric files covering 11 primary models and 2 smaller comparison models.
• —rubrics/rewrite/ the cascade-rewritten human rubric and its audit trail used in Finding 3.
• —finding1/rubric_as_response_capture/ the 100-case rubric-as-response capture check for Gemini 2.5 Pro, GPT-5.4, Opus 4.6, and the four smaller-model baselines.
• —finding1/open_ended_response_eval/ same-case human-rubric response-scoring artifacts for the smaller-model baselines.
• —finding2/coverage/ cosine-pool coverage artifacts (global_unique_t70/, human_model_unique_t70_all/).
• —finding2/direct_check/ the LLM-judge raw outputs and the dual-method intersection summaries used in the Finding 2 headline ratio.
• —finding2/normative_tendencies/ per-model same-branch normative-label summaries used in the Finding 2 label-share table.
• —finding2/normative_dimension_labels/ per-criterion label outputs used to build the same-branch summaries.
• —finding3/criterion_pairs/ the matched-pair file used to compute the Finding 3 fulfillment gap.
• —finding3/cascade_rescoring/ per-model rescoring outputs used to compute the Finding 3 rewrite-uplift table.
• —finding3/generality_validation/ cross-judge checks for the generality requirement.
• —canonical_full/ the full canonical rerun corpus.
• —responses/ model responses used to score every fulfillment judgement.
• —rubrics/ the per-model rubric corpus the responses are scored against.
• —criterion_match/ per-(model, case) matching pipeline outputs (intermediate; consumed by the Finding 1 build step).
• —answer_eval/ per-model judgement outputs under the original human rubric, the cascade-rewritten human rubric, and the model-generated rubric.
• —common100/ the 100-case slice shared across the matched-pair and coverage analyses.

Provenance and verification

• —paper_release/finding3/criterion_pairs/finding1_confirmed_pairs.json and paper_release/rubrics/rewrite/human_rubric_cascade_rewritten.jsonl are released as authoritative staged derived artifacts. The companion repository's code/validation/validate_release_metrics.py recomputes the Finding 1, Finding 2, and Finding 3 headline numbers directly from the released artifacts and writes a report under manifests/validation/metrics_report.json.
• —The companion repository's manifests/sha256.json lists hashes for the staged files that anchor the paper numbers.
• —reviewer_sample/manifest.json lists the files copied into the representative sample, with byte sizes and SHA-256 hashes.

License

MIT. See the companion code repository's LICENSE. Re-running the API-backed stages requires the user's own provider credentials and remains subject to the relevant model providers' terms of use.