LeoLakshman/ai-human-text-detector

AI vs. Human Text Detection

🔗 Live app: huggingface.co/spaces/LeoLakshman/ai-human-text-detector

Problem: Given a piece of text (pasted or uploaded as PDF/DOCX), decide whether it was written by a human or generated by an AI language model, and explain why.

Project 1 built 6 trained classifiers (classical ML + deep learning) for this task. Project 2 extends that app with two Large Language Models that add functionality the classical models can't provide on their own: an independent second opinion, and a plain-English explanation of the verdict.

Dataset

train_data_with_labels.xlsx → 8,176 documents, perfectly balanced (4,088 human / 4,088 AI), label 0 = human, label 1 = AI. Split 80/20 stratified into data/training_data/train.csv and data/test_data/test.csv.

Models used

Machine/deep learning (Project 1, unchanged): SVM (98.0% acc.), Decision Tree (91.5%), AdaBoost (94.8%), sklearn FNN (99.1%), plus Keras LSTM/CNN trained on Word2Vec embeddings. See "Models" below for where each is stored.

LLMs (Project 2, new) — two models, two distinct jobs: | Model | Role | Why this model | |---|---|---| | Qwen2.5-0.5B-Instruct (988MB) | Independent AI-vs-human classifier. Reads the raw text and makes its own judgment + one-line reason, shown alongside the classical model's verdict as a cross-check. | Small (0.5B params), ungated, fast enough for free CPU Spaces. | | SmolLM2-360M-Instruct (724MB) | Explanation generator. Given the classical/DL model's prediction plus its linguistic-feature evidence (sentence length, vocabulary richness, top SVM-weighted words, ...), writes a 2-3 sentence plain-English explanation grounded in those numbers. | Chat-tuned, ungated; picked specifically for its small footprint (vs. TinyLlama-1.1B-Chat's 2.2GB) to keep the combined LLM download under ~1.7GB. |

Both run locally inside the app, each in its own short-lived subprocess (see utils/llm_worker.py) — this app also loads TensorFlow in-process for the Keras LSTM/CNN models, and TensorFlow + PyTorch deadlock if both are doing real work in the same process, so each LLM call runs isolated instead. The LLM Insights tab can be toggled off in the sidebar if you want instant, LLM-free predictions.

What's in here

ai_human_detection_project/
├── app.py                      # Streamlit web app
├── Dockerfile                  # Hugging Face Spaces (Docker SDK) entrypoint
├── requirements.txt
├── README.md
├── .gitignore
├── utils/
│   ├── text_features.py        # shared linguistic-feature extractor (used by both notebook & app)
│   ├── remote_models.py        # downloads large model artifacts from the GitHub Release
│   ├── llm_utils.py             # Project 2 — LLM cross-check + explanation generation
│   └── llm_worker.py            # Project 2 — subprocess entry point for LLM calls
├── models/                      # trained model artifacts (see "Models" below)
├── data/
│   ├── training_data/train.csv
│   └── test_data/test.csv
├── scripts/                     # pipeline scripts (feature engineering, training, tuning)
└── notebooks/
    └── project1_notebook.ipynb  # Project 1 — all 4 required sections

Setup

python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt

Models

Committed directly to the repo (small enough for normal git):

• —models/tfidf_vectorizer.pkl, models/linguistic_scaler.pkl
• —models/svm_model.pkl — SVM, 98.0% accuracy
• —models/decision_tree_model.pkl — Decision Tree, 91.5% accuracy
• —models/adaboost_model.pkl — AdaBoost, 94.8% accuracy
• —models/fnn_sklearn_model.pkl — feedforward neural network (sklearn MLPClassifier), 99.1% accuracy — fast reference FNN

Hosted as [GitHub Release assets](https://github.com/LeoLakshman/ai_human_detection_project/releases/tag/v1.0.0) instead of committed directly — the LSTM and CNN models are each ~18MB, over GitHub's comfortable size limit for a normal push:

• —fnn_model.h5, lstm_model.h5, cnn_model.h5 — trained in Google Colab (GPU)
• —word2vec.model, tokenizer.json — embeddings/tokenizer needed by those models

app.py downloads these automatically on first run via utils/remote_models.py and caches them in models/ afterward, so you don't need to fetch them by hand. If you re-train and re-upload new versions, bump the tag in utils/remote_models.py (GITHUB_RELEASE_BASE_URL) to match.

Running the app

streamlit run app.py

The model selector only lists models that successfully loaded (or downloaded) — so it still works even if a release asset is temporarily unavailable, just with a smaller set of models to choose from. The first time you check "Run LLM cross-check + explanation" in the sidebar, it downloads Qwen2.5-0.5B-Instruct and SmolLM2-360M-Instruct (~1.7GB total) from the Hugging Face Hub and caches them; uncheck that box for instant LLM-free predictions.

Notebook

notebooks/project1_notebook.ipynb contains all four required sections. Sections 1–2 and the classical-ML half of Section 3 (SVM, Decision Tree, AdaBoost, sklearn FNN) run end-to-end locally. The Keras FNN/LSTM/CNN were trained in Colab (scripts/03_deep_learning_RUN_LOCALLY.py has the training code) — their results are merged into the notebook's Section 4 comparison table via deep_learning_results.json.

Deploying to Hugging Face Spaces

This Space uses the Docker SDK (a Dockerfile in the repo root runs the Streamlit app on port 7860) rather than Spaces' built-in Streamlit runtime — it gives full control over the Python environment, which this app needs to set USE_TF=0 before any LLM library import (see "A note on TensorFlow + PyTorch" below).

1. 1.Create a Space at huggingface.co/new-space, SDK = Docker, hardware = free CPU basic.
2. 2.Push this repo's contents to the Space's git remote (the YAML block at the top of this README configures the Space — title, SDK, port, etc.):

   git remote add space https://huggingface.co/spaces/<your-username>/<space-name>
   git push space main --force

--force is needed because Hugging Face seeds a new Space with its own placeholder commit (a starter Dockerfile/app.py) that has unrelated history to this repo — force-pushing replaces that placeholder with this project. It does not affect anything else.

1. 1.The LSTM/CNN weights download automatically on first run (from the GitHub Release) and are cached in the Space's persistent storage afterward.
2. 2.The two LLMs are handled differently: the Dockerfile pre-downloads them during the image build itself (not on a live user request), so the first build takes several minutes (PyTorch + TensorFlow + ~1.7GB of LLM weights baked into the image), but every app visit afterward — including the very first one — loads already-cached weights from disk instead of blocking a user's click on a multi-GB download.

A note on TensorFlow + PyTorch in the same process

This app uses TensorFlow (Project 1's Keras LSTM/CNN models) and PyTorch (Project 2's LLMs) side by side. Two separate issues showed up here, both from TensorFlow and PyTorch sharing one process:

• —transformers lazily imports TensorFlow internally to check available backends, and doing that in a process that already has PyTorch active deadlocks on a TensorFlow-internal mutex. Fixed by setting USE_TF=0 before transformers is imported (utils/llm_utils.py, top of file).
• —Even with that fix, once TensorFlow is already loaded in-process (this app loads it unconditionally at startup for the LSTM/CNN models) and PyTorch later does real work in the same process, they still deadlock — this surfaced as the live app hanging indefinitely on the first LLM call. Fixed by running every LLM call in its own short-lived subprocess (utils/llm_worker.py, invoked via llm_classify_subprocess / llm_explain_subprocess in utils/llm_utils.py) that never imports TensorFlow at all, instead of loading the LLM pipelines in-process.

A note on TensorFlow's Keras 3 H5 loader

The LSTM/CNN .h5 files were saved by a slightly different Keras 3.x minor version than what gets installed here, which can write a stray quantization_config key into layer configs that the installed version's layer constructors don't recognize. app.py's load_keras_models() patches keras.layers.Layer.from_config to strip that key generically before reconstructing any layer.

Results & what I learned

• —The classical/DL pipeline from Project 1 still drives the actual AI/Human prediction (SVM/FNN ≈ 98–99% test accuracy) — the LLMs were added as a second, independent layer, not a replacement.
• —The Qwen2.5-0.5B-Instruct cross-check usually agrees with the statistical models on clear-cut text, but disagreements are informative: they tend to surface on short or stylistically ambiguous passages where the TF-IDF/ linguistic-feature signal is weak — exactly where a second opinion is most useful.
• —Grounding the explanation in the document's own linguistic-feature values (rather than letting it free-generate) was necessary — ungrounded prompts produced plausible-sounding but generic explanations that didn't actually reflect the specific document.
• —Small (sub-1B) instruct/chat models are sufficient for both jobs here and keep the app deployable on a free CPU Space; nothing in this app requires GPU-scale models. Going from TinyLlama-1.1B (2.2GB) to SmolLM2-360M (724MB) for the explainer noticeably shrank the deployment with some loss in explanation nuance — an acceptable tradeoff at this scale.