yaoandy107/greyscope-qwen3.5-4b

Greyscope (Qwen3.5-4B)

This model is a `unsloth/Qwen3.5-4B-Base` model finetuned for AI-text detection on the EditLens dataset. It classifies text as human-written, AI-edited, or AI-generated, and loads with plain transformers.

English-only for now; Traditional Chinese and Japanese are planned for v2.

Repository: `yaoandy107/greyscope`

Model details

• —The task is ternary AI-text detection (human / AI-edited / AI-generated), plus a continuous 0–1 score for the degree of AI involvement.
• —The head is AutoModelForSequenceClassification with 4 buckets over edit magnitude; the bf16 LoRA (r=32) is merged into the base.
• —The license is CC BY-NC-SA 4.0, research and non-commercial only, inherited from the dataset.
• —The task and data follow the EditLens paper (arXiv:2510.03154).

The 4-bucket distribution is decoded to a 0–1 score by a weighted average; two validation-calibrated thresholds (shipped in calibration.json) split it into human / AI-edited / AI-generated.

Uses

• —Intended use: flagging likely AI-written or AI-edited English text, with a 0–1 score so you can set your own threshold.
• —Out of scope: it is not a substitute for human judgment. Don't use it as sole evidence in high-stakes decisions like academic integrity or employment.

How to use

Requires transformers>=5.5.0 (Qwen3.5 architecture support).

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "yaoandy107/greyscope-qwen3.5-4b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()
model.config.pad_token_id = tok.pad_token_id or tok.eos_token_id

This loads the raw model, which outputs 4 bucket logits. The calibrated decode to a human / AI-edited / AI-generated label and 0–1 score (using calibration.json) is in greyscope/inference.py.

Weights are ~9 GB in bf16. The calibrated thresholds are tuned for bf16; re-validate them if you load another dtype or quantization.

Evaluation

Greyscope is evaluated against the open detectors from the OpenPangram blog, on the same splits and protocol. It leads in-domain and on the unseen generator, ties editlens-Llama on Enron, and has the lowest false-positive rate on non-native English; editlens-Llama leads on RAID.

In-domain (ternary, n=6,115)

Held-out domain: Enron (ternary, n=6,147)

Held-out generator: Llama-70B (ternary, n=5,957)

RAID (TPR at 5% FPR, n=10,000)

Scored with RAID's fixed-FPR protocol (per-domain, 5% FPR) on its non-adversarial 10k sample. The OpenPangram blog reports macro-F1, but that leaves detectors at different false-positive rates, so the scores aren't comparable; a detector can rank higher just by flagging more humans.

Human-Detectors (binary, n=300)

Non-native English (humans only, n=91), FPR (lower is better):

Footprint

Warm forward pass on an M1 Pro (MPS, bf16, batch=1; median of 20, one-time model load excluded):

Limitations and biases

• —Research and non-commercial use only (CC BY-NC-SA 4.0).
• —English only; Traditional Chinese and Japanese are planned for v2.
• —Least reliable on lightly-edited and out-of-domain text.
• —The default threshold favors few false accusations (~1% even on non-native English); raise it if you need more recall.

Training

A single bf16 LoRA run on Qwen3.5-4B-Base with a 4-bucket sequence-classification head, about 4 hours on one A100-80GB. The task and data follow EditLens.

Citation

@article{Thai2025EditLens,
  title   = {EditLens: Quantifying the Extent of AI Editing in Text},
  author  = {Thai, Katherine and Emi, Bradley and Masrour, Elyas and Iyyer, Mohit},
  journal = {arXiv preprint arXiv:2510.03154},
  year    = {2025}
}

Acknowledgements

• —Open Pangram — the EditLens paper, open dataset, and open-source code this model learned from and builds on.
• —Modal — training ran on their free monthly compute credits.
• —Unsloth — efficient LoRA fine-tuning.