Tania-Amanda/ai-human-detection-projecthf

AI vs Human Text Detection

CS-5331: Introduction to Large Language Models Texas Tech University | Summer I 2026

Project Overview

This project trains and compares six machine learning and deep learning classifiers to detect whether a piece of text was written by a human or generated by AI. Project 2 extends the original Project 1 pipeline by integrating two Large Language Models that add explanation and independent judgment capabilities, and moves deployment from Streamlit Community Cloud to Hugging Face Spaces.

The original classifier pipeline remains fully functional. Nothing from Project 1 was removed, only extended.

Live app: https://huggingface.co/spaces/Tania-Amanda/ai-human-detection-projecthf

Models

Classifiers (Project 1)

Three traditional ML models and three deep learning models were trained on a labeled dataset of 8,176 samples (perfectly balanced between human and AI text).

Traditional ML: Support Vector Machine, Decision Tree, AdaBoost

Deep Learning: Feedforward Neural Network, LSTM, CNN for Text

Best performing model: FNN at 99.33% test accuracy.

LLMs (Project 2)

Two Qwen2.5 models were integrated, each with a distinct and justified role rather than overlapping functionality.

Qwen2.5-1.5B-Instruct — Explainer. Takes the classifier's prediction, confidence score, top TF-IDF terms, and linguistic feature summary, and generates a plain-English explanation of why the text was classified the way it was. This translates statistical output into something a non-technical user can understand.

Qwen2.5-0.5B-Instruct — Independent Judge. Given only the raw text, with no information about what the classifiers decided, this model makes its own AI-vs-human call with a short justification. The app then compares this independent verdict against the classifier's prediction and flags agreement or disagreement.

This setup allows for a genuine comparison between statistical ML classification and LLM-based judgment on the same text, which is discussed further in the notebook analysis.

Project Structure

ai_human_detection_project/
├── app.py                        # Streamlit app (local / Streamlit Cloud version)
├── requirements.txt              # Dependencies
├── README.md
├── models/
│   ├── svm_model.pkl
│   ├── decision_tree_model.pkl
│   ├── adaboost_model.pkl
│   ├── fnn_model.h5
│   ├── lstm_model.h5
│   ├── cnn_model.h5
│   ├── tfidf_vectorizer.pkl
│   ├── scaler.pkl
│   └── tokenizer.pkl
├── data/
│   ├── training_data/train.csv
│   └── test_data/test.csv
└── notebooks/
    └── ai-human-detection-project1.ipynb

The Hugging Face Spaces deployment uses a separate Docker-based repository with the app at src/streamlit_app.py, since HF Spaces' Streamlit template is Docker-based rather than a standalone SDK.

Setup and Installation (Local)

1. Clone the repository

git clone https://github.com/yourusername/ai_human_detection_project.git
cd ai_human_detection_project

2. Create and activate a conda environment

conda create -n ai_detection python=3.11
conda activate ai_detection

3. Install dependencies

pip install -r requirements.txt

4. Run the app

streamlit run app.py

The app will open at http://localhost:8501.

The first time the LLM Analysis section runs, it downloads both Qwen models from Hugging Face (a few GB total). After the first run they are cached locally.

Usage

• —Paste text directly into the text box or upload a PDF or Word document
• —Select a model from the sidebar for the primary classifier prediction
• —Click Analyze Text
• —View confidence scores, linguistic feature breakdown, and TF-IDF term importance across all six classifiers
• —Read the LLM-generated explanation of the prediction
• —Compare the LLM's independent judgment against the classifier's verdict
• —Download a summary report as a .txt file, including both LLM outputs

Models

The trained classifier model files are not included in this repository due to file size. You have two options:

Option 1 — Run the notebook. Open notebooks/ai-human-detection-project1.ipynb in Kaggle or Jupyter, run all cells, and the models will be saved to the models/ directory automatically.

Option 2 — Download from Kaggle. The trained models are available as output files from the Kaggle notebook. Download them and place them in the models/ directory before running the app.

The LLM models (Qwen2.5-1.5B-Instruct and Qwen2.5-0.5B-Instruct) are downloaded automatically from Hugging Face on first run and do not need to be manually provided.

Feature Engineering

Three feature representations were implemented and compared:

TF-IDF — 10,000 features with unigrams and bigrams. Most effective representation overall.

GloVe Embeddings — Pre-trained 100-dimensional vectors from the 2024 Wikipedia and Gigaword corpus. Used by LSTM and CNN.

Linguistic Features — 11 handcrafted stylistic signals including average word length, type-token ratio, punctuation density, and Flesch readability scores.

Model Performance

Known Limitations

The classifiers were trained on academic and essay-style text. Performance may degrade on other text types such as legal documents, social media posts, or technical writing that stylistically differs from the training distribution. During testing, formal documents like terms of service agreements were sometimes misclassified as AI-generated due to their repetitive, formal structure resembling common AI writing patterns.

The independent LLM judge (Qwen2.5-0.5B) occasionally disagrees with the trained classifiers, particularly on short or ambiguous inputs. This is expected and is treated as a finding rather than a bug, since it highlights the difference between statistical pattern matching and language model judgment on the same task.

The LLM components run on CPU by default and add a few seconds of latency per analysis compared to the classifier-only pipeline.

Challenges and How They Were Solved

Python and dependency version mismatches. Hugging Face Spaces defaulted to a newer Python version than TensorFlow supports, which caused the build to fail outright. Pinning Python to 3.11 in the Dockerfile fixed this. A similar issue showed up with transformers and torch, an unpinned install pulled incompatible versions and broke the package's internal imports. Pinning specific known-compatible versions (torch==2.5.1, transformers==4.46.3, accelerate==1.1.1) resolved it.

Hugging Face Spaces uses a Docker-based Streamlit template, not a standalone SDK. This meant the app file had to live at src/streamlit_app.py rather than a root-level app.py, and the Dockerfile had to explicitly copy the models/ and .streamlit/ directories into the image, since anything not listed in a COPY instruction simply doesn't exist in the container at runtime.

File upload returning a 403 error on Spaces. Streamlit's default XSRF and CORS protections conflicted with HF's reverse proxy setup. Disabling enableXsrfProtection and enableCORS in .streamlit/config.toml fixed the upload path without compromising the rest of the app.

Git LFS rejecting large files. Both the training dataset and the original Excel file exceeded GitHub and Hugging Face's file size limits for regular git tracking. The first few attempts at adding LFS tracking failed because the large file had already been committed in an earlier, non-LFS commit still present in the branch history, so git kept trying to push the old blob alongside the new one. The fix was git reset --soft origin/main to roll the branch pointer back without losing local changes, then re-adding .gitattributes LFS rules before staging the large files again in a single clean commit.

The independent LLM judge initially gave inconsistent or off-topic answers. With sampling enabled (do_sample=True), the smaller 0.5B model sometimes failed to commit to a single verdict or generated reasoning unrelated to the actual input text. Switching to greedy decoding (do_sample=False) made the output format and reasoning far more reliable.

Small LLMs sometimes disagree with the trained classifiers. Rather than treating this as a bug, the app surfaces these disagreements directly, since they highlight a real and interesting gap between statistical pattern-based classification and LLM judgment on the same input.

Deployment

This project is deployed in two places:

Streamlit Community Cloud — runs the classifier-only pipeline from app.py, no LLM integration.

Hugging Face Spaces — full Project 2 deployment including both LLMs, running via Docker with the Streamlit SDK template. Link above.

Dependencies

See requirements.txt for the full list. Main libraries: Streamlit, scikit-learn, TensorFlow, NLTK, Gensim, textstat, pdfplumber, python-docx, transformers, torch, accelerate.