Apppublic

abeldhe/ai-human-text-detector

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes

App README

AI vs. Human Text Detection — with LLMs (Project 2)

An app that predicts whether a piece of text was written by a human or generated by AI, combining classic machine-learning / deep-learning models with three Large Language Models, deployed on Hugging Face Spaces.

🔗 Live app: https://huggingface.co/spaces/abeldhe/ai-human-text-detector

Problem

AI-generated text is now everywhere in academic and professional settings, and telling it apart from human writing is a real, unsolved problem. This project builds a detector, makes its decision explainable, and stress-tests it by trying to fool it — so a user can not only get a verdict but understand and probe it.

Dataset

~8,200 labeled text samples, perfectly balanced (4,088 human, 4,088 AI), averaging ~325 words each. Split 80/20 into train/test, stratified to keep the class balance. The human samples range from informal student essays to academic prose; the AI samples are model-generated passages.

Models used

Machine / deep learning (from Project 1): six classifiers trained on TF-IDF features and GloVe-initialized sequences — SVM, Decision Tree, AdaBoost (classic ML) and FNN, LSTM, CNN (deep learning). Best test accuracy ~98% (FNN) and ~97% (SVM).

Large Language Models (Project 2) — three, each with a distinct purpose:

#	LLM	Role	Why it's used
1	`Qwen/Qwen2.5-1.5B-Instruct`	Detector — independent AI/Human second opinion	An LLM "votes" alongside the ML model so you can see agreement/disagreement. The 1.5B model clearly outperformed the 0.5B version, which was biased toward "Human" — a concrete example of model size improving output.
2	`TinyLlama/TinyLlama-1.1B-Chat-v1.0`	Explainer — plain-English reasoning	Turns the verdict into a human-readable explanation of vocabulary, structure, and tone — something the ML model alone can't produce.
3	`Qwen/Qwen2.5-0.5B-Instruct`	Humanizer — rewrites the text	Rewrites the text to sound more human, then re-scores it, showing whether the detector can be fooled (the cat-and-mouse reality of AI detection).

The LLMs run in bfloat16 so all three fit within the free Space's memory, and they load lazily (only when the LLM option is enabled).

Results & what I learned

—TF-IDF was by far the strongest feature (96% alone vs. 77% GloVe, 68% linguistic) — AI-vs-human detection is mostly about word choice.
—The simplest models won: FNN and linear SVM beat the heavier LSTM/CNN and trained far faster.
—For the LLMs, bigger meant better: the 1.5B detector reliably distinguished AI from human text where the 0.5B model failed.
—The detector reaches ~97% on in-distribution text but over-predicts AI on out-of-distribution text — a real limitation, which is exactly why the humanizer and explanation features matter. This is a demonstrator of why detection is hard, not a tool to accuse anyone.

Fine-tuning (extra credit)

Instead of relying only on the zero-shot LLM detector (which guesses AI vs. human), I fine-tuned a model specifically trained for the task:

—Base model: distilbert-base-uncased
—Dataset: `Ateeqq/AI-and-Human-Generated-Text` — 28,662 balanced samples (real vs. GPT-3-generated academic abstracts); trained on 12,000, evaluated on 3,000.
—Training: 2 epochs on Apple-Silicon GPU (MPS), ~17 minutes. Script: finetune_detector.py.
—Result: 99.8% test accuracy / 0.998 F1.

The fine-tuned model is shown as its own highlighted section in the app, next to the zero-shot LLM verdict, so you can compare a trained detector against one that only guesses.

What this taught me: the fine-tuned model is dramatically more confident and accurate within its training domain (academic-style text), but because it was trained only on abstracts it can mislabel out-of-domain text (e.g., a product listing). That's the core lesson — fine-tuning gives big gains on the distribution it sees, but doesn't magically generalize everywhere.

Run locally

bash

pip install -r requirements.txt
streamlit run app.py

The app: paste text or upload a .pdf/.docx/.txt, pick a model, and get a prediction with confidence, influential words, text statistics, a six-model comparison, a fine-tuned DistilBERT detector, the three-LLM analysis, and a downloadable report.