Modelpublic

frankhanhj/roberta-base-all-sources-ai-identifier

sourceHugging Faceotherupdated 4mo agoView on Hugging Face

0likes180downloads

Model Card

RoBERTa All-Sources AI Text Identifier

This model is a roberta-base sequence classifier fine-tuned to distinguish human-written text from AI-generated text, with emphasis on creative-writing and prose-like distributions.

Labels:

—human: label 0
—ai: label 1

Intended Use

The model is intended for research and internal evaluation of AI-generated prose detection. It should not be used as the sole basis for high-stakes decisions about authorship, academic integrity, employment, moderation, or user sanctions. Outputs are classifier probabilities, not proof of authorship.

Loading

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "frankhanhj/roberta-base-all-sources-ai-identifier"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

Training Data

The training corpus is datasets/all-sources-ai-identifier, assembled from the local sources below. Validation and test rows were not used for training. For chunked external sources, splits were isolated by group_id / original_index; the recorded split audit found 0 leaks across 10,992 checked groups.

Source	Train Rows	Validation Rows	Test Rows	Labels / Notes
MAGE / HC3 / RAID mixed research corpus	10,000	1,600	1,600	Balanced human/AI mixture from `hc3`, `mage`, and `raid`
RL Claude creative-writing SFT outputs	3,549	442	446	AI-only generated writing
Gale Creativewriting GPT outputs	23,986	2,843	2,960	AI-only generated creative writing
`human-writing-dpo`	21,501	2,670	2,629	`chosen_human` as human, `rejected_ai` as AI
Human creative MAGE holdout	800	100	100	Human-only `roct_human` and `wp_human`

Overall split sizes:

Split	Rows	Human	AI
Train	59,836	18,322	41,514
Validation	7,655	2,473	5,182
Test	7,735	2,401	5,334

Techniques Tried

Several detector families were trained or evaluated before selecting this checkpoint:

Run	Technique	Train Rows	Test Rows	Test Accuracy	Test ROC AUC
`roberta-base-all-sources-ai-identifier`	RoBERTa sequence classifier on all sources	59,836	7,735	0.984874	0.998999
`roberta-base-mixed-plus-claude-ai-identifier`	RoBERTa sequence classifier on mixed research + Claude	16,000	1,600	0.991250	0.999239
`roberta-base-mixed-research-ai-identifier`	RoBERTa sequence classifier on MAGE/HC3/RAID	10,000	1,600	0.989375	0.999022
`token-qwen35-mixed-research-ai-identifier`	Qwen3.5 token-level CNN/attention classifier	10,000	1,600	0.981875	0.998786
`qwen35-2b-mixed-research-embedding-identifier`	Pooled Qwen3.5 embeddings + MLP	10,000	1,600	0.971875	0.998188
`raw-mixed-research-ai-identifier`	Raw text n-gram + stylometric classifier	10,000	1,600	0.965625	0.989867
`raw-mixed-plus-claude-ai-identifier`	Raw text n-gram + stylometric classifier on mixed + Claude	16,000	1,600	0.950000	0.985572
`qwen35-2b-mage-embedding-identifier`	Pooled Qwen3.5 embeddings + MLP on MAGE	40,000	6,000	0.861500	0.939538
`roberta-base-mage-4k-ai-identifier`	RoBERTa sequence classifier on small MAGE subset	4,000	1,000	0.738000	0.883386
`token-qwen35-mage-4k-ai-identifier`	Qwen3.5 token-level CNN/attention classifier on small MAGE subset	4,000	1,000	0.772000	0.864189
`raw-mage-ai-identifier`	Raw text n-gram + stylometric classifier on MAGE	40,000	6,000	0.730000	0.813246

The selected model is the all-sources RoBERTa checkpoint. It has slightly lower original mixed-benchmark accuracy than the earlier mixed-plus-Claude RoBERTa run, but it fixes a severe false-positive failure on human-writing-dpo.

Training Procedure

Base model: roberta-base

Key settings:

—Max sequence length: 512
—Batch size: 8
—Evaluation batch size: 64
—Gradient accumulation: 2
—Epochs: 2
—Learning rate: 2e-5
—Weight decay: 0.01
—Warmup ratio: 0.06
—Class weighting: balanced
—Mixed precision: fp16
—Seed: 20260530
—Best checkpoint selection: validation ROC AUC

The selected checkpoint is epoch 1. Epoch 2 increased AI recall slightly but reduced human recall, so epoch 1 was kept.

Evaluation

All-sources held-out test result:

Accuracy	ROC AUC	Human Recall	AI Recall	Confusion Matrix
0.984874	0.998999	0.967097	0.992876	`[[2322, 79], [38, 5296]]`

Comparison against the previous mixed-plus-Claude RoBERTa model on the all-sources test set:

Model	Accuracy	ROC AUC	Human Recall	AI Recall
all-sources RoBERTa	0.984874	0.998999	0.967097	0.992876
previous mixed-plus-Claude RoBERTa	0.792889	0.706478	0.433153	0.954818

Important held-out slice results:

Test Slice	Accuracy	Human Recall	AI Recall
`human-writing-dpo`	0.977178	0.960027	1.000000
Gale Creativewriting GPT	0.989527	n/a	0.989527
RL Claude Creative Writing SFT	1.000000	n/a	1.000000
Human creative MAGE	0.940000	0.940000	n/a
MAGE mixed	0.973333	0.960000	0.986667
HC3	0.997500	0.995000	1.000000
RAID	0.995000	1.000000	0.990000

Known Limitations

—The model is sensitive to the distributions represented in the assembled datasets.
—AI text detectors can produce false positives, especially under domain shift.
—The training set is not balanced by source; balanced class loss was used, but source imbalance remains.
—Some evaluation slices are single-label, so they report only human recall or AI recall rather than full binary metrics.
—The model should be calibrated and re-evaluated before use on new domains, languages, or generator families.

Artifacts

This repository contains the full fine-tuned Transformers checkpoint:

—model.safetensors
—config.json
—tokenizer.json
—tokenizer_config.json
—metrics.json
—history.json
—run_config.json
—group_metrics.json