MahatirTusher/bangla-ai-text-detector

🇧🇩 Bengali AI-Generated Text Detector (BanglaBERT-SupCon v2)

    

A state-of-the-art, cross-generator resilient sequence classification model for detecting AI-generated Bengali text.

Fine-tuned on BanglaBERT (csebuetnlp/banglabert) via Supervised Contrastive Learning (SupCon) and calibrated using Temperature Scaling ($T = 1.8816$), this model is specifically engineered to overcome generator-shift vulnerabilities. It reliably catches machine-generated Bengali text across frontier LLMs including ChatGPT, Gemini, Claude, and DeepSeek while strictly preserving genuine human Bengali writing.

📑 Table of Contents

1. 1.Core Scientific Contributions & Why SupCon
2. 2.Architectural Variants & Ablation Study (Method A–E)
3. 3.Multi-Seed Reproducibility & LOGO Evaluation
4. 4.Comparative Analysis vs. Baselines & Existing Detectors
5. 5.Overcoming the Limitations of Prior Bengali Detectors
6. 6.Quick Start & Inference Guide
7. 7.Operating Modes & Decision Thresholds
8. 8.Dataset & Diagnostic Sanity Checks
9. 9.Citation & Author Attribution

🔬 Core Scientific Contributions & Why SupCon

Standard transformer cross-entropy fine-tuning suffers from generator overfitting: models memorize surface-level artifacts, repetitive syntactic templates, and generator-specific phrasing (e.g., DeepSeek's paragraph formatting or Claude's introductory style) rather than intrinsic machine-synthesized semantics. Consequently, traditional detectors experience catastrophic failure when exposed to unseen LLMs.

🌟 Why Supervised Contrastive Learning (SupCon)?

To achieve true generator invariance, our architecture integrates Supervised Contrastive Loss ($$\mathcal{L}_{\text{SupCon}}$$) with Label-Smoothing Cross-Entropy ($\mathcal{L}_{\text{LS-CE}}$):

$$\mathcal{L}{\text{total}} = \mathcal{L}{\text{LS-CE}} + \lambda \mathcal{L}_{\text{SupCon}}$$

1. 1.Latent Manifold Clustering: SupCon pulls all AI representations (regardless of whether synthesized by OpenAI, Google, Anthropic, or DeepSeek) into a tightly bounded, cohesive hyperspherical cluster, while simultaneously repelling authentic human text in latent space.
2. 2.Zero-Shot Transfer on Unseen LLMs: When an unobserved generator is tested under Leave-One-Generator-Out (LOGO) conditions, the network classifies it based on its core structural AI signatures rather than memorized generator fingerprints.
3. 3.Multi-Seed Stability: Delivers near-zero variance across random initializations and consistent 94%+ zero-shot recall.

       [ Human Writing Space ] <===================> [ Universal AI Manifold (SupCon) ]
  (Poetry, Formal, News, Blogs)    Margin Repulsion     ├── ChatGPT (GPT-5.6 Luna)
                                                        ├── Claude (Sonnet 4)
                                                        ├── DeepSeek (DeepSeek-V4)
                                                        └── Google (Gemini 3.1 Pro)

🧪 Architectural Variants & Ablation Study

To isolate the source of generalizability, we benchmarked 5 distinct architectural paradigms (Methods A through E) under identical Leave-One-Generator-Out (LOGO) protocols:

• —Method A (`A_standard_baseline`): Standard BanglaBERT (Cross-Entropy) + Validation Threshold Optimization.
• —Method B (`B_capacity_control`): BanglaBERT with Layer-wise LR decay ($0.8$), Weight Decay ($0.05$), and Dropout ($0.25$).
• —Method C (`C_hybrid_char_lsa`): Multi-modal fusion of BanglaBERT embeddings + Character $n$-gram Latent Semantic Analysis (LSA).
• —Method D (`D_balanced_sampler`): Class-Balanced Dynamic BatchSampler ($8\text{ Human} + 8\text{ AI}$ per optimization step).
• —Method E (`E_supcon` - OUR PROPOSED MODEL): Supervised Contrastive Learning + Temperature Scaling Calibration.

Ablation Ladder Summary (LOGO Cross-Validation)

Conclusion: While Method C artificially inflated recall by memorizing character $n$-grams, it degraded human specificity down to ~78% on DeepSeek holdouts (falsely accusing authentic Bengali writers). Method E (SupCon) achieved the only true optimal balance—high invariant recall without penalizing human authors.

📊 Multi-Seed Reproducibility & LOGO Benchmark

To guarantee scientific reproducibility, Method E (SupCon) was evaluated across 3 random seeds ($42, 123, 2024$) across all 4 LOGO folds ($12$ full training and evaluation cycles).

Per-Generator Mean $\pm$ Standard Deviation across Multi-Seed Runs

<details> <summary><b>🔍 Click to view the granular Seed-by-Seed Fold Breakdown (Seeds 42, 123, 2024)</b></summary>

</details>

🥊 Comparative Analysis vs. Baselines & Existing Detectors

We benchmarked our model against baseline paradigms and empirical competitors on identical unseen holdouts:

Paired Bootstrap Significance Test (Our Model vs. B2 on Unseen Folds)

• —On ChatGPT Holdout: $+4.75\%$ gain ($p = 0.0000$, $95\%\text{ CI: } [2.98\%, 6.40\%]$)
• —On Gemini Holdout: $+15.79\%$ gain ($p = 0.0000$, $95\%\text{ CI: } [13.28\%, 18.24\%]$)
• —On DeepSeek Holdout: $+8.53\%$ gain ($p = 0.0000$, $95\%\text{ CI: } [6.67\%, 10.45\%]$)
• —On Claude Holdout: $+5.39\%$ gain ($p = 0.0000$, $95\%\text{ CI: } [3.95\%, 6.67\%]$)

🛡️ How We Overcame Limitations of Prior Bengali Detectors

🚀 Quick Start & Inference Guide

1. Minimal pipeline Interface

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="MahatirTusher/bangla-ai-text-detector",
    return_all_scores=True
)

text = "ভিটেলোজেনিন (Vitellogenin বা Vtg) জিন বিবর্তনের একটি অত্যন্ত গুরুত্বপূর্ণ দিক।"
predictions = classifier(text)

print(predictions)
# Output: [[{'label': 'Human', 'score': 0.0178}, {'label': 'AI', 'score': 0.9822}]]

2. PyTorch Inference with Calibrated Temperature Scaling (Recommended)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_ID = "MahatirTusher/bangla-ai-text-detector"
TEMPERATURE = 1.8816

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.eval()

def detect_bengali_ai(text: str, threshold: float = 0.50):
    inputs = tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        max_length=256
    )
    
    with torch.no_grad():
        logits = model(**inputs).logits[0]
        # Apply empirical temperature calibration
        calibrated_logits = logits / TEMPERATURE
        probs = torch.softmax(calibrated_logits, dim=-1)
        
    human_prob = float(probs[0].item())
    ai_prob = float(probs[1].item())
    
    verdict = "AI-generated" if ai_prob >= threshold else "Human-written"
    
    return {
        "verdict": verdict,
        "ai_probability": round(ai_prob, 4),
        "human_probability": round(human_prob, 4),
        "confidence": "very_high" if abs(ai_prob - threshold) >= 0.35 else "high" if abs(ai_prob - threshold) >= 0.20 else "moderate"
    }

# Example Test
sample = "বাংলাদেশ দক্ষিণ এশিয়ার একটি নদীমাতৃক ও সার্বভৌম রাষ্ট্র।"
print(detect_bengali_ai(sample))

🎯 Operating Modes & Decision Thresholds

🔍 Dataset & Diagnostic Integrity

• —Total Dataset: 15,000 strictly balanced Bengali texts ($7,500\text{ Human} + 7,500\text{ AI}$).
• —Generators: 4 Frontier LLMs ($1,875$ samples each from ChatGPT, Gemini, DeepSeek, Claude).
• —Topic Taxonomy: 45 diverse subjects (Science, Economics, Journalism, Archaeology, Philosophy, Tech, etc.).
• —Lexical Leakage Prevention: Split performed at (generator_model, topic) group level—no semantic group straddles train/val/test.
• —Near-Duplicate Check: Jaccard 5-gram overlap between train and test is only $2.71\%$, confirming zero sentence memorization.

👨‍💻 Author & Citation

• —Principal Investigator & Author: Mahatir Ahmed Tusher
• —AI Data Generator: Sagar Chandra Dey
• —Initiative: Khoj Project — Advanced Fact-Checking & AI Content Verification
• —Base Model: csebuetnlp/banglabert

@misc{tusher2025bengaliaidetector,
  author = {Mahatir Ahmed Tusher, Sagar Chandra Dey},
  title = {Bengali AI-Generated Text Detector via Supervised Contrastive Learning (BanglaBERT-SupCon v2)},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/MahatirTusher/bangla-ai-text-detector}}
}

📄 License

This model and its artifacts are distributed under the MIT License.