HoopitAI/video-deepfake-detection-GenD_CLIP_L_14_FF

Video Deepfake Detection — video-deepfake-detection-GenDCLIPL14FF

State-of-the-art Video Deepfake Detection model trained on FaceForensics++ (FF++) based on the GenD framework fine-tuned with Sharpness-Aware Minimization (SAM) and Label Smoothing.

This model uses openai/clip-vit-large-patch14 as visual foundation backbone and fine-tunes only the Layer Normalization parameters (accounting for only ~0.03% of total parameters) while enforcing a hyperspherical feature manifold through L2-normalization and metric learning (Uniformity & Alignment losses).

📊 Benchmark Results

📌 Model Details

• —Training Dataset: FaceForensics++ (FF++) (FF++)
• —Visual Backbone: openai/clip-vit-large-patch14
• —Classification Head: LinearNorm
• —Optimizer: SAM-AdamW (SAM $\rho=0.05$, adaptive=True)
• —Loss Formulation: Cross-Entropy with Label Smoothing (0.1), Uniformity (0.5), Alignment (0.1)
• —Training Epochs: 30
• —Batch Size: 96
• —Precision: bf16-mixed
• —Learning Rate: 0.0003

🚀 Quickstart & Inference

1. Using the Model in Python

import torch
from PIL import Image
from transformers import AutoModel

# Load the model directly from Hugging Face Hub
model = AutoModel.from_pretrained("HoopitAI/video-deepfake-detection-GenD_CLIP_L_14_FF", trust_remote_code=True)
model.eval()

# Preprocess image crop (aligned face)
image = Image.open("path/to/face_crop.png").convert("RGB")
tensor = model.feature_extractor.preprocess(image).unsqueeze(0)

# Run inference
with torch.no_grad():
    logits = model(tensor)
    # Output class 0: Real, Output class 1: Fake
    fake_prob = logits.softmax(dim=-1)[0, 1].item()

print(f"Deepfake Probability: {fake_prob:.2%}")

2. Using with src.hf.modeling_gend

from src.hf.modeling_gend import GenD

model = GenD.from_pretrained("HoopitAI/video-deepfake-detection-GenD_CLIP_L_14_FF")
model.eval()

🏗️ Architecture & Training Methodology

The GenD method achieves superior cross-dataset generalization by avoiding catastrophic overfitting on manipulation-specific artifacts:

1. 1.Training on FF++: Trained on face crops from FaceForensics++ (FF++).
2. 2.LayerNorm Tuning: Keeps the visual transformer backbone frozen while updating only normalization scaling and bias terms.
3. 3.Normalized Linear Head: Normalizes feature vectors onto a hypersphere before linear projection.
4. 4.SAM Optimization: Smooths the loss landscape to find flat minima that resist out-of-distribution domain shifts.