Arko007/deepfake-detector-dfd-sota

SPDX-License-Identifier: Apache-2.0

""" SOTA Deepfake Detector - DFD Model card + inference utilities for Hugging Face repository: Arko007/deepfake-detector-dfd-sota

This file contains:

• —A model card in Hugging Face model README format (YAML frontmatter + sections).
• —Inference helper functions to:
• —load frames or an MP4 video,
• —extract 12 evenly-spaced frames,
• —preprocess using the CLIP image processor,
• —run a forward pass through the provided checkpoint and return logits,
• —return softmax probabilities and predicted class.

Notes:

• —Architecture described: CLIP-ViT-Large (frozen backbone) + Spatiotemporal Adapters
• —Input: 12 frames, 3x384x384, dtype=torch.float32 / BF16 where supported
• —Labels: 0 = real, 1 = fake
• —License: Apache-2.0
• —This file is intended as a convenience reference for model consumers and deployers. """

SOTA Deepfake Detector - DFD

Model: SOTA Deepfake Detector - DFD Repository: https://huggingface.co/Arko007/deepfake-detector-dfd-sota

Model Description

The "SOTA Deepfake Detector - DFD" is a spatiotemporal adaptation built on a frozen CLIP-ViT-Large backbone with lightweight spatiotemporal adapters inserted to learn temporal relationships across frames. The backbone parameters are frozen; only the adapters and final classification head are trainable.

• —Architecture: CLIP-ViT-Large (frozen backbone) + Spatiotemporal Adapters
• —Input resolution: 384×384 pixels
• —Temporal frames: 12 frames per video
• —Trainable parameters: 5,255,938 (≈1.7% of total 308M)
• —Precision used for training: BF16 (mixed precision)
• —Framework: PyTorch + Transformers (Hugging Face)
• —HF Hub repo: Arko007/deepfake-detector-dfd-sota
• —Model files:
• —pytorch_model.bin (1.28GB)
• —config.json

Intended Use

This model is intended to classify short video clips (or frame sequences) as either:

• —Label 0: Real videos
• —Label 1: Deepfake/manipulated videos

Primary use-cases:

• —Research on deepfake detection
• —Benchmarking against other detectors
• —Integration into content-moderation pipelines with caution

Not intended:

• —Medical, legal, or other high-stakes decisions without human review.
• —Use on domains/styles substantially different from the DFD dataset without further fine-tuning.

Training Data

Dataset: DFD (Deep Fake Detection)

• —Total videos: 3,431
• —Real videos: 363 (10.6%)
• —Fake videos: 3,068 (89.4%)
• —Preprocessing:
• —12 evenly-spaced frames extracted per video at 384×384 resolution (total frames: 41,172)
• —Train/validation split: 90/10 stratified
• —Class balancing during training: WeightedRandomSampler (inverse frequency weights)

Training Configuration

• —Batch size: 16 (effective 32 with gradient accumulation)
• —Optimizer: AdamW (weight_decay=0.05)
• —Learning rate: 5e-6 with cosine decay, 10% warmup
• —Epochs: 12
• —Loss: Cross-entropy with label smoothing = 0.1
• —Gradient clipping: maxgradnorm = 1.0
• —Sampling: WeightedRandomSampler to mitigate class imbalance
• —Augmentation (training only): random horizontal flip, brightness jitter
• —Hardware: NVIDIA L4 GPU (24GB VRAM)
• —Random seed: 42 (all RNGs fixed for reproducibility)

Training speed: ~1.4 seconds per batch (batch_size=16) on the reported hardware.

Evaluation / Metrics

• —Best validation accuracy: 84.88%
• —Validation detection (approx. ranges due to small real class):
• —Real class detection: ~47–55% (low number of real samples)
• —Fake class detection: ~64–93%
• —Training loss convergence (cross-entropy w/ label smoothing): 0.7097 → 0.6921 (12 epochs)

Known Limitations

• —Validation set is highly imbalanced (89% fake), which affects stability of metrics.
• —Small number of real videos (363) limits generalization to unseen real samples.
• —Model optimized for the DFD dataset; transfer to other deepfake types may require fine-tuning.
• —Temporal context limited to 12 frames (approx. 0.4–1s depending on FPS), so long-term artifacts may be missed.

Usage

Quick inference instructions:

1. 1.Load checkpoint: checkpoint = torch.load("bestmodel.pt", maplocation="cpu")
2. 2.Extract model state: modelstate = checkpoint["modelstate_dict"]
3. 3.Initialize model: model = DeepfakeDetector(config)
4. 4.Load state dict: model.loadstatedict(model_state)
5. 5.Set to eval: model.eval()
6. 6.Inference: Pass a tensor of shape (batch_size, 12, 3, 384, 384) with dtype float32 (or bf16 where supported).
7. 7.Output:
8. 8.logits: (batch_size, 2)
9. 9.probabilities: softmax(logits)
10. 10.prediction: argmax(logits) -> 0=real, 1=fake

How to Cite

If you use this model in your work, please cite the repository and include details about the DFD dataset and this specific model configuration.

"""

----------------------------

Inference code

----------------------------

The following is a compact, self-contained inference utility. It assumes:

- checkpoint is saved as 'best_model.pt'

- a config object or config.json for the model is available

- CLIP image processor is available via transformers

#

Important: This implementation is a minimal example to run inference. For

production, wrap in a robust server, add batching, async IO, error handling,

and pre-warming for BF16 on supported accelerators.

import os import math from typing import List, Tuple, Union, Dict

import numpy as np from PIL import Image import torch import torch.nn.functional as F from torchvision import transforms from torchvision.io import read_video # optional, requires torchvision installation from transformers import CLIPImageProcessor, CLIPModel

----------------------------------------------------------------------

Model definition stub

----------------------------------------------------------------------

The real model used for training is a CLIP-ViT-Large backbone (frozen) with

spatiotemporal adapters and a small classification head. For inference the

exact architecture must match the checkpoint. Below is a minimal class

to demonstrate expected load / inference semantics. Replace this with the

model class used during training (and the one saved to the checkpoint).

class DeepfakeDetector(torch.nn.Module): """ Minimal wrapper around CLIP backbone + temporal adapters + classification head.

NOTE: This is a lightweight placeholder for demonstration. Replace with the exact model definition used during training to successfully load the provided checkpoint (pytorchmodel.bin / bestmodel.pt). """

def _init(self, clipmodelname: str = "openai/clip-vit-large-patch14", numframes: int = 12): super()._init() self.numframes = numframes # Load CLIP and freeze it (backbone frozen) self.clip = CLIPModel.frompretrained(clipmodelname, torchdtype=torch.float32) for p in self.clip.parameters(): p.requiresgrad = False

# Spatiotemporal adapters & head (trainable) # NOTE: The real training used custom adapters; here we provide a representative head. embeddim = self.clip.visualprojection.shape[1] if hasattr(self.clip, "visualprojection") else self.clip.config.projectiondim self.adapterpool = torch.nn.AdaptiveAvgPool1d(1) # placeholder # small trainable head consistent with ~5.26M params total self.classifier = torch.nn.Sequential( torch.nn.Linear(embeddim, 1024), torch.nn.ReLU(), torch.nn.Dropout(p=0.1), torch.nn.Linear(1024, 2) )

def forward(self, x: torch.Tensor) -> torch.Tensor: """ Forward pass.

x: Tensor of shape (batch, num_frames, 3, H, W) Returns logits: (batch, 2) """ b, t, c, h, w = x.shape # reshape to (bt, c, h, w) to run through CLIP's visual encoder xt = x.view(b t, c, h, w)

# CLIP's image encoder expects pixel values and image processing externally. # Use clip.visionmodel to obtain image embeddings outputs = self.clip.visionmodel(pixelvalues=xt) pooled = outputs.pooleroutput # (b*t, embed_dim) or adjust per CLIP impl

# reshape back to (b, t, embeddim) and aggregate temporally embeddim = pooled.shape[-1] pooled = pooled.view(b, t, embeddim) # (b, t, embeddim)

# Simple temporal aggregation (real model uses adapters). For inference placeholder: # mean across temporal dimension videorepr = pooled.mean(dim=1) # (b, embeddim)

logits = self.classifier(video_repr) # (b, 2) return logits

----------------------------------------------------------------------

Helper utilities for frame extraction, preprocessing and inference

----------------------------------------------------------------------

def extractevenlyspacedframesfromvideo( videopath: str, numframes: int = 12, targetsize: Tuple[int, int] = (384, 384), ) -> List[Image.Image]: """ Extract num_frames evenly spaced frames from a video file.

Returns a list of PIL Image objects resized to targetsize. Requires torchvision.readvideo; as a fallback, uses ffmpeg via PIL + imageio if readvideo is unavailable. """ if not os.path.exists(videopath): raise FileNotFoundError(f"Video file not found: {video_path}")

try: # torchvision's readvideo returns (frames, audio, info) frames, , info = readvideo(videopath, ptsunit="sec") # frames: (numtotalframes, H, W, C) uint8 tensor total = frames.shape[0] if total == 0: raise RuntimeError("No frames extracted from video.") indices = np.linspace(0, total - 1, numframes, dtype=int) pilframes = [] for i in indices: frame = frames[i].numpy() img = Image.fromarray(frame) img = img.convert("RGB").resize(targetsize, resample=Image.BILINEAR) pilframes.append(img) return pilframes except Exception: # Fallback: use imageio-ffmpeg or other mechanism (not implemented here) raise RuntimeError("Video reading failed. Ensure torchvision is installed and supports read_video.")

def loadframesfromfolder( folder: str, numframes: int = 12, targetsize: Tuple[int, int] = (384, 384), ) -> List[Image.Image]: """ Load frames (PNG/JPG) from a folder. Picks numframes evenly across available images. """ files = sorted( [ os.path.join(folder, f) for f in os.listdir(folder) if f.lower().endswith((".png", ".jpg", ".jpeg")) ] ) if not files: raise FileNotFoundError(f"No image files found in folder: {folder}") total = len(files) indices = np.linspace(0, total - 1, numframes, dtype=int) pilframes = [] for i in indices: img = Image.open(files[i]).convert("RGB").resize(targetsize, resample=Image.BILINEAR) pilframes.append(img) return pil_frames

def preprocessframes( pilframes: List[Image.Image], processor: CLIPImageProcessor, device: Union[str, torch.device] = "cpu", ) -> torch.Tensor: """ Given a list of PIL images (len == 12), apply the CLIP image processor and return a tensor shaped (1, 12, 3, 384, 384) suitable for the model. """ # processor can batch process a list of images; it returns dict with pixelvalues # pixelvalues shape: (numimages, 3, H, W) proc = processor(images=pilframes, returntensors="pt") pixelvalues = proc["pixelvalues"] # (numimages, 3, H, W) # Ensure exactly numframes pixelvalues = pixelvalues.to(device) # reshape to (1, numframes, 3, H, W) pixelvalues = pixelvalues.unsqueeze(0) if pixelvalues.ndim == 4 else pixelvalues pixelvalues = pixelvalues.view(1, len(pilframes), 3, pixelvalues.shape[-2], pixelvalues.shape[-1]) return pixelvalues

def applyaugmentationstrain(pilframes: List[Image.Image]) -> List[Image.Image]: """ Apply training augmentations: horizontal flip (random) and brightness jitter. Called only during training data pipeline; included here for completeness. """ aug = transforms.Compose([ transforms.RandomHorizontalFlip(p=0.5), transforms.ColorJitter(brightness=0.2), ]) return [aug(img) for img in pilframes]

----------------------------------------------------------------------

High level inference API

----------------------------------------------------------------------

def loadmodelfromcheckpoint( checkpointpath: str, clipmodelname: str = "openai/clip-vit-large-patch14", device: Union[str, torch.device] = "cpu", ) -> Tuple[torch.nn.Module, CLIPImageProcessor]: """ Load model and CLIP image processor.

checkpoint should be a dict with 'modelstatedict' key (per model card instructions). """ device = torch.device(device) # Instantiate model skeleton model = DeepfakeDetector(clipmodelname=clipmodelname, num_frames=12)

# Load checkpoint ckpt = torch.load(checkpointpath, maplocation="cpu") if "modelstatedict" in ckpt: statedict = ckpt["modelstatedict"] else: # assume full state dict saved directly statedict = ckpt

# Try to load, allowing for missing keys if adapter names differ try: model.loadstatedict(state_dict) except Exception as e: # Provide a clearer error for mismatch: user must use exact model implementation raise RuntimeError(f"Failed to load state dict into DeepfakeDetector: {e}")

model.to(device) model.eval()

# Load CLIP image processor (matches model's backbone preprocessing) processor = CLIPImageProcessor.frompretrained(clipmodel_name) return model, processor

def predictfromframes( model: torch.nn.Module, processor: CLIPImageProcessor, pilframes: List[Image.Image], device: Union[str, torch.device] = "cpu", usebf16: bool = False, ) -> Dict[str, Union[int, float, List[float], torch.Tensor]]: """ Perform inference on a single sample (list of 12 PIL frames). Returns a dict with:

• —logits (torch.Tensor shape (1,2))
• —probabilities (list of 2 floats)
• —prediction (int, 0 or 1)
• —confidence (float, probability of predicted class) """ device = torch.device(device) x = preprocessframes(pilframes, processor, device=device) # (1,12,3,384,384) # optionally convert to bf16 if on supported hardware if use_bf16 and device.type != "cpu": x = x.to(dtype=torch.bfloat16) else: x = x.to(dtype=torch.float32) model = model.to(device)

with torch.nograd(): logits = model(x) # (1,2) probs = F.softmax(logits, dim=-1) probslist = probs.squeeze(0).cpu().tolist() pred = int(torch.argmax(logits, dim=-1).squeeze().item()) confidence = float(probs.squeeze(0)[pred].cpu().item())

return { "logits": logits.cpu(), "probabilities": probs_list, "prediction": pred, "confidence": confidence, }

----------------------------------------------------------------------

Convenience function: video file -> prediction

----------------------------------------------------------------------

def predictfromvideofile( checkpointpath: str, videopath: str, device: Union[str, torch.device] = "cpu", clipmodelname: str = "openai/clip-vit-large-patch14", numframes: int = 12, usebf16: bool = False, ) -> Dict[str, Union[int, float, List[float], torch.Tensor]]: """ Load model from checkpoint, extract frames from video, and return prediction. """ model, processor = loadmodelfromcheckpoint(checkpointpath, clipmodelname=clipmodelname, device=device) pilframes = extractevenlyspacedframesfromvideo(videopath, numframes=numframes, targetsize=(384, 384)) return predictfromframes(model, processor, pilframes, device=device, usebf16=usebf16)

----------------------------------------------------------------------

Example usage (commented)

----------------------------------------------------------------------

if _name == "main_":

# Example: run inference on a single MP4

checkpoint = "best_model.pt"

video = "example.mp4"

result = predictfromvideofile(checkpoint, video, device="cuda", usebf16=True)

print("Logits:", result["logits"])

print("Probabilities:", result["probabilities"])

print("Prediction (0=real,1=fake):", result["prediction"], "confidence:", result["confidence"])

----------------------------------------------------------------------

End of file

----------------------------------------------------------------------