suraj-ranganath/tell-human-detectors

TELL Human Detectors Split

This dataset is a prompt-disjoint validation/test split of human_detectors.json from Jenna Russell, Marzena Karpinska, and Mohit Iyyer, "People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text" (arXiv:2501.15654).

The split is intended as a research artifact for evaluating AI-vs-human writing detectors and explanation-quality methods. It preserves the upstream fields, with one documented issue type normalization for Hugging Face compatibility, and adds split provenance metadata.

This TELL split artifact was prepared by Suraj Ranganath and Aldan Creo. The underlying Human Detectors dataset should be cited to Russell, Karpinska, and Iyyer.

A subset of this was used in the paper: arxiv.org/abs/2605.27921

Source

• —Upstream repository: https://github.com/jenna-russell/human_detectors
• —Source file: https://raw.githubusercontent.com/jenna-russell/human_detectors/refs/heads/main/human_detectors.json
• —Source dataset license: MIT
• —Canonical source SHA256: 1b8f9eeb6413f2541a320b09c7cc77b8d5c51a9e69a87ae0e91bd0e9ede5bd9a

Split Protocol

The original dataset has 300 documents, 30 prompt_id groups, and 10 rows per prompt group. Each prompt group contains exactly one row for each combination of ground_truth in AI-generated, Human-written and generation_model in claude, gpt-4o, humanized_o1-pro, o1-pro, and paraphrased_gpt-4o.

We split by prompt_id, not by individual row, to prevent the same underlying prompt/topic from appearing in both validation and test.

• —Split seed: tell-human-detectors-v1-2242
• —Validation prompt IDs: [4, 8, 13, 14, 15, 16, 20, 23, 25, 26]
• —Test prompt IDs: [1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30]
• —Validation rows: 100
• —Test rows: 200

Both splits are balanced by label and generation model. Validation has 50 human-written and 50 AI-generated documents. Test has 100 human-written and 100 AI-generated documents.

Fields

All upstream fields are preserved, including article metadata, article, ground_truth, automatic detector outputs, five annotator objects, majority_vote, and expert_majority_vote. The upstream issue field mixes strings and integers, so this artifact stores issue as a string for Hugging Face/Arrow compatibility and records the original Python type in issue_original_type.

Additional fields:

• —split: validation or test
• —source_row_index: integer row index in the sorted upstream JSON object
• —source_dataset: upstream dataset identifier
• —source_url: upstream source URL
• —source_sha256: canonical upstream checksum used by this artifact
• —split_method: prompt_id_grouped_sha256_seed
• —split_seed: deterministic split seed
• —issue_original_type: original upstream JSON type for issue

Recommended Use

Use validation for prompt, metric, and model-selection decisions. Use test once for final reporting. Do not tune on the test set, and do not merge validation and test without reporting that choice.

Citation

@misc{russell2025humandetectors,
  title={People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text},
  author={Russell, Jenna and Karpinska, Marzena and Iyyer, Mohit},
  year={2025},
  eprint={2501.15654},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}