Modelpublic

Ef05/ai-text-detection-summaries-xlm-r-v1

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

Model Card

Multilingual AI Text Detector (Short Summaries)

CLI scripts for training, evaluating, and running a two-stage AI text detection system for short multilingual summaries.

—Stage A: Human vs AI detection (calibrated probability output)
—Stage B: Generator attribution over {chatgpt, gemini, claude, llama, deepseek} (calibrated probability output), conditional on Stage A

Designed for GPU environments with simple pip installation. No Docker or complex dependencies required.

Documentation

—[docs/QUICKSTART.md](docs/QUICKSTART.md) — Get started in 5 minutes
—[docs/CLI.md](docs/CLI.md) — Comprehensive CLI usage guide
—[configs/default.yaml](configs/default.yaml) — Full configuration reference

Quick Start

bash

# Install dependencies
pip install -r requirements.txt

# Train a model
python train.py --config configs/default.yaml --run-name my_first_model

# Evaluate
python evaluate.py --run-dir runs/my_first_model

# Predict
python predict.py --run-dir runs/my_first_model --text "Your text here" --lang en

See docs/QUICKSTART.md for detailed setup instructions.

End-to-End Workflow

The typical full-pipeline sequence:

bash

# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# (Only if using LLaMA route and the checkpoint requires auth)
huggingface-cli login
huggingface-cli whoami

# Scan data inventory and infer schema report
python scan_data.py --data-root data --out runs/data_scan.json --preview-rows 25

# (Optional) Write dataset artefacts (clean/dedup/splits) for inspection
python -m src.data.build_dataset --config configs/default.yaml --out-dir runs/datasets/latest

# Train full two-stage pipeline (including LLaMA QLoRA if enabled in config)
python train.py --config configs/default.yaml --run-name my_run

# Evaluate (writes runs/<run>/metrics.json + plots under runs/<run>/reports/)
python evaluate.py --run-dir runs/my_run

# Thesis plots (writes outputs/plots/<run_id>/...)
python -m src.evaluation.evaluate_run --run_id my_run --stage A
python -m src.evaluation.evaluate_run --run_id my_run --stage B

# Prediction and explanation examples
python predict.py --run-dir runs/my_run --text "..." --lang en
python explain.py --run-dir runs/my_run --text "..." --lang en --out reports/explanations/example_en

Cross-run comparison:

bash

python -m src.evaluation.evaluate_run --compare runs:<id1>,<id2>,<id3>

Re-run only the LLaMA QLoRA track (using existing splits in a run directory):

bash

python train_llama.py --config configs/default.yaml --run-dir runs/my_run --stage both

Core Commands

Data Preparation

Scan data directory for CSV files and infer schema:

bash

python scan_data.py --data-root data --out runs/data_scan.json

Training

Train two-stage detector with calibration:

bash

python train.py --config configs/default.yaml

Options:

—--run-name: Custom name for output directory
—--run-dir: Resume from existing run
—--log-level: Set logging verbosity (DEBUG, INFO, WARNING)

Evaluation

Evaluate trained model on test set with metrics and visualisations:

bash

python evaluate.py --run-dir runs/latest

Produces:

—Classification metrics (accuracy, F1, precision, recall)
—Calibration metrics (ECE, Brier score)
—Confusion matrices and reliability diagrams

Evaluation Plots

The evaluation package generates thesis-ready plots for each stage and supports cross-run comparisons.

Per-run, Stage A (binary):

bash

python -m src.evaluation.evaluate_run --run_id <RUN_ID> --stage A

Per-run, Stage B (multiclass):

bash

python -m src.evaluation.evaluate_run --run_id <RUN_ID> --stage B

Compare multiple runs:

bash

python -m src.evaluation.evaluate_run --compare runs:<id1>,<id2>,<id3>

Outputs are written to outputs/plots/<run_id>/stage_A/ and outputs/plots/<run_id>/stage_B/ (plus outputs/plots/compare/ for comparisons). Defaults are configured in configs/eval.yaml.

Prediction

Predict on a single text with confidence scores:

bash

python predict.py \
  --run-dir runs/latest \
  --text "Text to analyze" \
  --lang en

Options:

—--text-file: Read text from file instead
—--stage-a-model: Choose Stage A model (auto, transformer, hybrid, llama)
—--stage-b-model: Choose Stage B model (auto, transformer, llama)
—--explain-level: Control explanation detail (off, basic, detailed)
—--force-stage-b: Always run Stage B even for human predictions

Explanation

Generate detailed attribution artefacts:

bash

python explain.py \
  --run-dir runs/latest \
  --text "Text to explain" \
  --lang en \
  --out reports/explanations/example

Creates visualisations and attribution scores for model interpretability.

Web UI (SLURM / HPC)

The web UI runs on a university HPC cluster via SSH tunnel. It requires three components: a GPU allocation (SLURM), uvicorn on the login node, and an SSH tunnel from the local machine.

1. Get a GPU allocation (SSH into the cluster):

bash

srun --partition=a2000-6h --gres=gpu:nvidia_rtx_a2000_12gb:1 \
     --cpus-per-task=2 --mem=8G --time=01:00:00 --pty bash -l

Note the job ID from squeue -u $USER in another terminal, then exit back to the login node (exit or Ctrl+D). The allocation stays running.

2. Run uvicorn on the login node (on the cluster, from the login node):

bash

cd ~/finale
source .venv/bin/activate
export UI_SRUN_JOBID=<JOBID>          # from squeue
export UI_PREDICT_PYTHON="$HOME/finale/.venv/bin/python3"
export UI_SRUN_ARGS="--overlap"
export RUN_DIR="runs/llama31bv10"
export TIMEOUT_SECONDS=1800
python3 -m uvicorn ui_app.app:app --host 127.0.0.1 --port 8000

3. SSH tunnel from local machine (Git Bash / terminal on your PC):

bash

ssh -N -L 8000:localhost:8000 -p 6767 finneye@login.ucrel-hex.scc.lancs.ac.uk

4. Open the UI: navigate to http://localhost:8000 in a browser.

To stop: Ctrl+C uvicorn, scancel <JOBID>, and Ctrl+C the SSH tunnel.

Project Structure

.
├── train.py                  # Main training entry point
├── train_llama.py            # Standalone LLaMA QLoRA training
├── evaluate.py               # Evaluation entry point
├── predict.py                # Prediction entry point
├── explain.py                # Explanation generation entry point
├── scan_data.py              # Data discovery tool
├── requirements.txt          # Python dependencies
│
├── configs/                  # YAML experiment configurations
│   ├── default.yaml
│   └── eval.yaml
│
├── src/                      # Core source package
│   ├── data/                 #   Data loading, cleaning, splitting
│   ├── features/             #   Stylometric feature extraction
│   ├── models/               #   Transformer, hybrid, TF-IDF, LLaMA QLoRA
│   ├── calibration/          #   Temperature scaling and metrics
│   ├── evaluation/           #   Evaluation pipeline and plotting
│   ├── explain/              #   Integrated Gradients, SHAP, highlighting
│   ├── plotting/             #   Shared figure utilities
│   └── utils/                #   Config, logging, seeding, run management
│
├── data/                     # Processed CSV datasets
│   └── processed/
│       ├── ai-generated/
│       └── human-generated/
│
├── runs/                     # Training outputs and run artefacts
├── outputs/                  # Evaluation plots and reports
├── reports/                  # Figures, explanations, results
│
├── scripts/                  # Auxiliary scripts
│   ├── data_collection/      #   API and LLaMA generation scripts
│   ├── slurm/                #   HPC job submission scripts
│   ├── make_dissertation_figures.py
│   └── prompt_robustness_test.py
│
├── tests/                    # Unit tests
├── ui_app/                   # FastAPI web UI (runs on HPC via SSH tunnel)
│
└── docs/                     # Documentation
    ├── CLI.md
    └── QUICKSTART.md

Features

—Multiple model architectures: XLM-RoBERTa, hybrid (transformer + stylometry), LLaMA QLoRA
—Calibrated predictions: Temperature scaling for reliable confidence scores
—Multilingual support: Trained on English, Spanish, French, Chinese, and Arabic
—Explainability: Integrated Gradients, SHAP, n-gram analysis
—Memory efficient: 8-bit optimisers, gradient checkpointing, configurable batch sizes
—Deterministic: Seeded splits and training for reproducibility

Configuration

Default settings in configs/default.yaml:

—Model: XLM-RoBERTa Large, 128 tokens, bf16 precision
—Training: 3 epochs, early stopping, 8-bit AdamW optimiser
—Data: Deduplication enabled, stratified splits (70/15/15)
—Calibration: Temperature scaling with 15 bins
—LLaMA QLoRA route: Llama-3.1-8B with 4-bit QLoRA (enabled by default; configurable via llama.enabled)

Copy and modify the config to experiment with different settings.

Requirements

—Python 3.8+
—PyTorch 2.0+
—Transformers 4.30+
—CUDA-capable GPU (12 GB+ VRAM recommended for training)
—See requirements.txt for full dependencies

Expected Data Format

Place CSV files in the data/ directory. The system auto-discovers and infers schemas.

Preferred columns:

—Text: summary, text, content
—Label: is_ai, label, source_type (0 = human, 1 = AI)
—Language: language, lang (en, es, fr, zh, ar)
—Generator: generator, model_name (chatgpt, claude, gemini, llama, deepseek)

Example:

csv

summary,is_ai,language,generator
"The study shows...",0,en,
"Research indicates...",1,en,chatgpt
"Los resultados demuestran...",1,es,gemini

Troubleshooting

Out of memory?

—Reduce batch_size: 1 and increase gradient_accumulation_steps in config
—Enable gradient_checkpointing: true
—Use 8-bit optimiser: optim: "adamw_bnb_8bit"

Training too slow?

—Disable LLaMA: set llama.enabled: false
—Use fp16/bf16 mixed precision
—Reduce epochs or use a smaller model

See docs/CLI.md#troubleshooting for more solutions.

Tests

Run the evaluation-metrics unit tests:

bash

python -m unittest discover -s tests -p "test_*.py"

Licence

This project was developed as part of a university dissertation. It is intended for academic and research use only.

Acknowledgements

Built with:

—Transformers (Hugging Face)
—PyTorch
—Captum (model interpretability)
—SHAP (explainability)