CoolFace
Apppublic

Sumit0098073e5/Universal-Deepfake-Detector

sourceHugging Facemitupdated 8mo agoView on Hugging Face
2likes
App README

🕵️‍♂️ The Universal Forgery Detector: A Journey into Deepfake Forensics

Detecting the undetectable: A Hybrid Deep Learning approach to identifying AI-Generated Images (Generative AI), Deepfakes, and Traditional Image Manipulation.

📖 Table of Contents

  1. 1.Project Motivation: The "Why"
  2. 2.The Problem: The "Generation Gap"
  3. 3.Dataset Odyssey: From Chaos to Curation
  4. 4.Architecture: The Hybrid Brain
  5. 5.The Training Chronicles: Strategy & Pivots
  6. 6.Phase 1: The Foundation
  7. 7.Phase 2: The "Sniper" Fine-Tuning
  8. 8.Phase 3: The Grand Unification
  9. 9.Challenges Faced & Engineering Solutions
  10. 10.The Final Pipeline: From Input to Verdict
  11. 11.Results & Metrics
  12. 12.Installation & Usage
  13. 13.Acknowledgments

🌟 Project Motivation: The "Why"

In the era of Midjourney v6, DALL-E 3, and Stable Diffusion XL, the line between reality and fiction has evaporated.

  • —The Threat: Malicious actors use AI to forge documents, create fake news, and impersonate public figures.
  • —The Gap: Traditional forgery detectors (built for Photoshop splicing) fail completely against AI generation. They look for "noise inconsistencies" that modern Diffusion models simply do not produce.
  • —Our Mission: To build a "Universal" detector—one capable of spotting the mathematical fingerprints of both old-school pixel manipulation and modern Generative AI, without being fooled by high-resolution resizing artifacts.

📉 The Problem: The "Generation Gap"

During our initial research, we discovered a critical flaw in existing solutions: The Generation Gap.

  1. 1.Old Detectors (ELA/Noise Analysis): Worked on Photoshop and early GANs (Generative Adversarial Networks) because those methods left distinct "checkerboard" artifact patterns.
  2. 2.Modern AI (Diffusion Models): Models like Midjourney work by "denoising" random Gaussian noise. The resulting images are statistically "clean." They have no checkerboard artifacts.
  3. 3.The Result: A standard detector sees a Midjourney image, sees "clean noise," and confidently predicts "Real."

We realized we couldn't just throw data at a ResNet. We needed a model that could "see" two things at once: the invisible noise (Forensics) and the visual weirdness (Semantics).

📂 Dataset Odyssey: From Chaos to Curation

Our data strategy was not linear. It was an iterative process of finding blind spots and patching them.

1. The Starting Point (The Old World)

We began with standard datasets to teach the model "Basic Forgery":

  • —COCO 2017 (20k images): Served as our ground truth for "Real" images.
  • —CASIA 2.0 (5k images): The gold standard for Photoshop splicing and tampering.
  • —Tiny-GenImage: A small collection of older GAN-generated images.

Critique: This dataset was unbalanced (only 5k edited vs 20k real) and outdated.

2. The "Modern" Pivot (The New World)

To catch DALL-E 3 and Midjourney, we had to hunt for specific datasets.

  • —ArtiFact (The Giant): A massive 31GB dataset. We could not load this into RAM.
  • —Strategy: We wrote a custom Path Scout script using os.walk to surgically extract only folders containing keywords: stable_diffusion, glide, latent_diffusion.
  • —Midjourney v6 Prompts: A specific dataset targeting the hyper-realistic texture of Midjourney.
  • —The "Gemini Patch": We manually generated 88 images using ChatGPT and Gemini to ensure our model wasn't blind to the specific tools we used for testing.

3. The Grand Unified Dataset

Final composition for the "Universal" Model:

  • —Real: 4,000 (COCO)
  • —Edited: 4,000 (CASIA Tp)
  • —AI: 4,532 (Balanced mix of Midjourney, DALL-E 3, and Stable Diffusion)

🧠 Architecture: The Hybrid Brain

We rejected standard CNNs (like ResNet50) because they focus too much on content (e.g., "is this a cat?") rather than authenticity (e.g., "is this cat's texture consistent?").

We built a Dual-Stream Hybrid Network:

Stream A: The "Semantic Eye" (Swin Transformer)

  • —Input: RGB Image.
  • —Architecture: swin_tiny_patch4_window7_224 (Pretrained on ImageNet).
  • —Role: Detects High-Level Visual Artifacts.
  • —Examples: Plastic skin smoothing, inconsistent lighting shadows, "melting" objects, logic errors (3 hands).
  • —Why Swin? Transformers use "Attention" mechanisms, making them excellent at spotting global inconsistencies that CNNs miss.

Stream B: The "Forensic Magnifier" (BayarConv + EfficientNet)

  • —Input: RGB Image -> Passed through a Bayar Constraint Layer.
  • —The Bayar Layer: A learnable High-Pass Filter. It forces the network to ignore the image content (the dog) and look only at the pixel noise residuals.
  • —Backbone: efficientnet_b0.
  • —Role: Detects Low-Level Frequency Artifacts.
  • —Examples: JPEG grid disruptions, splicing edges, GAN checkerboard patterns.

The Fusion Head

  • —We concatenate the features from Stream A (Visual) and Stream B (Noise).
  • —Fully Connected Layers -> Softmax -> 3 Classes: [Real, Edited, AI].

📜 The Training Chronicles: Strategy & Pivots

Phase 1: The Foundation

  • —Goal: Train a baseline model on Real vs. Old AI vs. Edited.
  • —Technique: Used WeightedRandomSampler to handle the severe class imbalance (5k Edited vs 20k Real).
  • —Outcome: 95% Accuracy on test set.
  • —Failure: The model failed completely on "Modern" AI (Midjourney). It predicted them as "Real" because they lacked GAN artifacts.

Phase 2: The "Sniper" Fine-Tuning (Surgical Strike)

  • —Goal: Teach the model only about Diffusion Models (Modern AI).
  • —Strategy:
  • —Freeze Stream B (Noise): We locked the forensic layer. Why? Because Modern AI is "clean." If we trained Stream B, it would get confused by the lack of noise.
  • —Unfreeze Stream A (Visual): We allowed the Swin Transformer to learn the specific visual "glitches" of DALL-E and Midjourney.
  • —Outcome: The model learned to detect Midjourney with 98% accuracy!
  • —Disaster: Catastrophic Forgetting. When we tested it on old Photoshop edits, accuracy dropped to 42%. The model overwrote its knowledge of "Editing" to make room for "AI."

Phase 3: The Grand Unification (Replay Training)

  • —Goal: Restore balance.
  • —Strategy:
  • —Unfreeze ALL layers.
  • —Create a perfectly balanced dataset (4k Real / 4k Edited / 4k AI).
  • —Use a Very Low Learning Rate (`1e-5`) to gently adjust weights without destroying previous knowledge.
  • —Final Outcome:
  • —Real Accuracy: 99%
  • —Edited Accuracy: 96%
  • —AI Accuracy: 98%

⚔️ Challenges Faced & Engineering Solutions

1. The "Tiny Image" Trap

  • —Problem: When testing on small images (e.g., 200x200), standard resizing to 224x224 caused upscaling blur. The model misinterpreted this blur as "AI Artifacts," leading to False Positives.
  • —Solution: Implemented Smart Preprocessing.
  • —If Image < 224px: Pad with borders (don't stretch).
  • —If Image > 224px: Resize.

2. The DALL-E 3 Resolution Failure

  • —Problem: High-res ChatGPT images (1024x1024) looked "Real" to the model.
  • —Root Cause: Resizing a 1024px image to 224px destroys the microscopic artifacts DALL-E leaves behind (hair blending, texture glitches).
  • —Solution: The "Magnifying Glass" (Patch-Based Voting).
  • —Instead of resizing the whole image, we extract 5 crops (Center + 4 Corners) at full resolution.
  • —We run inference on each patch.
  • —Logic: If any patch is >90% AI, the whole image is flagged. This improved detection on DALL-E 3 from 25% to 80%.

3. The "Uncanny Valley" of High-Res Real Photos

  • —Problem: Extremely high-quality, bright, noise-free photos from modern phones were sometimes flagged as AI because they were "too perfect" (lacking sensor noise).
  • —Solution: Uncertainty Thresholding.
  • —If the model's top confidence score is < 60%, we explicitly label it "⚠️ Inconclusive" instead of forcing a guess.

🚀 The Final Pipeline: From Input to Verdict

  1. 1.Input: User uploads an image (any size).
  2. 2.Preprocessing:
  3. 3.Check Dimensions.
  4. 4.Apply Padding if small.
  5. 5.Apply Patch Extraction if large (Sliding Window Scan).
  6. 6.Inference:
  7. 7.Pass image/patches through Hybrid Model.
  8. 8.Stream A analyzes visual semantics.
  9. 9.Stream B analyzes noise residuals.
  10. 10.Fusion Layer combines vectors.
  11. 11.Logic Engine:
  12. 12.Aggregate scores from all patches.
  13. 13.Check against Confidence Threshold (60%).
  14. 14.Check against "Critical Artifact" Threshold (90% on single patch).
  15. 15.Output:
  16. 16.Label: Real / Edited / AI.
  17. 17.Confidence: % Score.
  18. 18.Reasoning: "Majority of patches look synthetic" or "Global structure looks natural."

📊 Results & Metrics

Test Set Performance (150 Unseen Images): | Class | Precision | Recall | F1-Score | | :--- | :--- | :--- | :--- | | Real | 0.96 | 1.00 | 0.98 | | Edited | 1.00 | 0.96 | 0.98 | | AI | 0.98 | 0.98 | 0.98 | | Overall | 0.98 | 0.98 | 0.98 |

Real World Stress Test:

  • —Successfully detected Midjourney v6 (Webp format).
  • —Successfully detected ChatGPT/DALL-E 3 (PNG format) using Patch Scanning.
  • —Correctly identified Photoshop Splicing (CASIA).

💻 Installation & Usage

Prerequisites

  • —Python 3.8+
  • —CUDA capable GPU (Recommended) or CPU

1. Clone the Repository

bash
git clone https://github.com/yourusername/universal-forgery-detector.git
cd universal-forgery-detector

2. Install Dependencies

bash
pip install -r requirements.txt
Note: On Linux systems, you may need to install libgl1: apt-get install libgl1-mesa-glx

3. Run the App

bash
streamlit run app.py

4. Docker (Optional)

bash
docker build -t forgery-detector .
docker run -p 8501:8501 forgery-detector

🤝 Acknowledgments

  • —CASIA Dataset Team — For the foundational forgery dataset.
  • —ArtiFact Project — For the comprehensive collection of modern AI generators.
  • —Timm Library — For the pristine Swin Transformer implementation.
  • —Albumentations — For the robust image augmentation pipeline. ---

<p align="center">Built with ❤️ and ☕ by <strong>SUMIT KUMAR</strong></p>