mhamza-007/cvit_deepfake_detection

🔍 Convolutional Vision Transformer (CViT) for Deepfake Detection

The Convolutional Vision Transformer (CViT) is a hybrid architecture combining the powerful spatial feature extraction capabilities of CNNs with the long-range dependency modeling of Vision Transformers (ViT). This model is purpose-built for detecting deepfake videos and is trained on DFDC.

Model Architecture

1. Feature Learning (FL) Module - CNN Backbone

• —Composed of 17 convolutional operations.
• —Unlike traditional VGG architectures, FL focuses purely on feature extraction, not classification.
• —Accepts input of size 224 × 224 × 3 (RGB image).
• —Outputs a 512 × 7 × 7 feature map.
• —Contains 10.8 million learnable parameters.

2. Vision Transformer (ViT) Module

• —Receives CNN output (512 × 7 × 7) as its input.
• —Converts the 7×7 patches into a 1 × 1024 sequence using linear embedding.
• —Adds positional embeddings of shape (2 × 1024).
• —ViT Encoder uses:
• —Multi-Head Self Attention (MSA) with 8 attention heads.
• —MLP blocks with:
• —First linear layer of 2048 units.
• —Final linear layer of 2 units (binary classification: Fake / Real).
• —ReLU activation and Softmax for final probabilities.

🧪 Experimental Results

The CViT model was tested and evaluated across multiple deepfake datasets:

📊 FaceForensics++ Accuracy

Note: Poor performance on the FaceShifter dataset is attributed to the model's difficulty in learning subtle visual artifacts.

🧪 DFDC Evaluation

• —Unseen DFDC test videos: 400
• —Accuracy: 91.5%
• —AUC Score: 0.91

🧪 UADFV AUC Comparison

⚙️ Training Configuration

• —Loss Function: Binary Cross Entropy (BCE)
• —Optimizer: Adam
• —Learning Rate: 1e-4
• —Weight Decay: 1e-6
• —Batch Size: 32
• —Epochs: 50
• —Learning Rate Scheduler: Reduces LR by factor of 0.1 every 15 epochs
• —Normalization:
• —Mean: [0.485, 0.456, 0.406]
• —Std: [0.229, 0.224, 0.225]

🧪 Inference Setup

• —Input: 30 normalized facial images (per video)
• —Classification:
• —Uses log loss function to compute confidence.
• —Output is a probability y ∈ [0, 1]
• —0 < y < 0.5: Real
• —0.5 ≤ y ≤ 1: Fake
• —Log loss penalizes:
• —Random guesses
• —Confident but incorrect predictions

🛠 Inference Example

from huggingface_hub import hf_hub_download
import torch

# Download model
model_path = hf_hub_download(
    repo_id="mhamza-007/cvit_deepfake_detection",
    filename="cvit2_deepfake_detection_ep_50.pth"
)

# Load model (example)
model = torch.load(model_path, map_location='cpu')
model.eval()