Agents-MCP-Hackathon/mcp-deepfake-forensics

• —08/05/25: This repository is no longer maintained. For latest updates please follow [Borderless R&D](https://huggingface.co/buildborderless).
• —6/18/25: YES, we are aware that updates to the submission will likely result in a disqualification. It was never about the cash prize for us in the first place 😉 Good luck to all hackers!

The Detection Dilemma: The Degentic Games

The cat-and-mouse game between digital forgery and detection reached a tipping point early last year after years of escalating concern and anxiety. The most ambitious, expensive, and resource-intensive detection model was launched with actually impressive results. Impressive… for an embarassing two to three weeks.

Then came the knockout punches. New SOTA models emerging every few weeks, in every imaginageable domain -- image, audio, video, music. Generated images are now at a level of realism that to an untrained eye, its unable to discern if its real or fake. [TO-DO: Add Citation to the study]

And let's be honest: we saw this coming. When has humanity ever resisted accelerating technology that promises... interesting applications? As the ancients wisely tweeted: 🔞 drives innovation.

It's time for a reset. Quit crying and get ready. Didn't you hear? The long awaited Degentic Games is starting soon, and your model sucks.

Re-Thinking Detection

1. Shift away from the belief that more data leads to better results. Rather, focus on insight-driven and "quality over quantity" datasets in training.

• —Move Away from Terabyte-Scale Datasets: Focus on quality over quantity by curating a smaller, highly diverse, and labeled dataset emphasizing edge cases and the latest AI generations.
• —Active Learning: Implement active learning techniques to iteratively select the most informative samples for human labeling, reducing dataset size while maintaining effectiveness.

2. Efficient Model Architectures

• —Adopt Lightweight, State-of-the-Art Models: Explore models designed for efficiency like MobileNet, EfficientNet, or recent advancements in vision transformers (ViTs) tailored for forensic analysis.
• —Transfer Learning with Fine-Tuning: Leverage pre-trained models fine-tuned on your curated dataset to leverage general knowledge while adapting to specific AI image detection tasks.

3. Multi-Modal and Hybrid Approaches

• —Combine Image Forensics with Metadata Analysis: Integrate insights from image processing with metadata (e.g., EXIF, XMP) for a more robust detection framework.
• —Incorporate Knowledge Graphs for AI Model Identification: If feasible, build or utilize knowledge graphs mapping known AI models to their generation signatures for targeted detection.

4. Continuous Learning and Update Mechanism

• —Online Learning or Incremental Training: Implement a system that can incrementally update the model with new, strategically selected samples, adapting to new AI generation techniques.
• —Community-Driven Updates: Establish a feedback loop with users/community to report undetected AI images, fueling model updates.

5. Evaluation and Validation

• —Robust Validation Protocols: Regularly test against unseen, diverse datasets including novel AI generations not present during training.
• —Benchmark Against State-of-the-Art: Periodically compare performance with newly published detection models or techniques.

Core Roadmap

[x] Project Introduction [ ] Agents Released into Wild [ ] Whitepaper / Arxiv Release [ ] Public Participation

Functions Available for LLM Calls via MCP

This document outlines the functions available for programmatic invocation by LLMs through the MCP (Multi-Cloud Platform) server, as defined in mcp-deepfake-forensics/app.py.

1. full_prediction

Description

This function processes an uploaded image to predict whether it is AI-generated or real, utilizing an ensemble of deepfake detection models and advanced forensic analysis techniques. It also incorporates intelligent agents for context inference, weight management, and anomaly detection.

API Names

• —predict

Parameters

• —img (str): The input image to be analyzed, provided as a file path.
• —confidence_threshold (float): A value between 0.0 and 1.0 (default: 0.7) that determines the confidence level required for a model to label an image as "AI" or "REAL". If neither score meets this threshold, the label will be "UNCERTAIN".
• —rotate_degrees (float): The maximum degree by which to rotate the image (default: 0). If greater than 0, "rotate" augmentation is applied.
• —noise_level (float): The level of noise to add to the image (default: 0). If greater than 0, "add_noise" augmentation is applied.
• —sharpen_strength (float): The strength of the sharpening effect to apply (default: 0). If greater than 0, "sharpen" augmentation is applied.

Returns

• —img_pil (PIL Image): The processed image (original or augmented).
• —cleaned_forensics_images (list of PIL Image): A list of images generated by various forensic analysis techniques (ELA, gradient, minmax, bitplane). These include:
• —Original augmented image
• —ELA analysis (multiple passes)
• —Gradient processing (multiple variations)
• —MinMax processing (multiple variations)
• —Bit Plane extraction
• —table_rows (list of lists): A list of lists representing the model predictions, suitable for display in a Gradio Dataframe. Each inner list contains: Model Name, Contributor, AI Score, Real Score, and Label.
• —json_results (str): A JSON string containing the raw model prediction results for debugging purposes.
• —consensus_html (str): An HTML string representing the final consensus label ("AI", "REAL", or "UNCERTAIN"), styled with color.

2. noise_estimation

Description

Analyzes image noise patterns using wavelet decomposition. This tool helps detect compression artifacts and artificial noise patterns that may indicate image manipulation. Higher noise levels in specific regions can reveal areas of potential tampering.

API Name

• —tool_waveletnoise

Parameters

• —image (PIL Image): The input image to analyze.
• —block_size (int): The size of the blocks for wavelet analysis (default: 8, range: 1-32).

Returns

• —output_image (PIL Image): An image visualizing the noise patterns.

3. bit_plane_extractor

Description

Extracts and visualizes individual bit planes from different color channels. This forensic tool helps identify hidden patterns and artifacts in image data that may indicate manipulation. Different bit planes can reveal inconsistencies in image processing or editing.

API Name

• —tool_bitplane

Parameters

• —image (PIL Image): The input image to analyze.
• —channel (str): The color channel to extract the bit plane from. Possible values: "Luminance", "Red", "Green", "Blue", "RGB Norm" (default: "Luminance").
• —bit_plane (int): The bit plane index to extract (0-7, default: 0).
• —filter_type (str): A filter to apply to the extracted bit plane. Possible values: "Disabled", "Median", "Gaussian" (default: "Disabled").

Returns

• —output_image (PIL Image): An image visualizing the extracted bit plane.

4. ELA

Description

Performs Error Level Analysis to detect re-saved JPEG images, which can indicate tampering. ELA highlights areas of an image that have different compression levels.

API Name

• —tool_ela

Parameters

• —img (PIL Image): Input image to analyze.
• —quality (int): JPEG compression quality (1-100, default: 75).
• —scale (int): Output multiplicative gain (1-100, default: 50).
• —contrast (int): Output tonality compression (0-100, default: 20).
• —linear (bool): Whether to use linear difference (default: False).
• —grayscale (bool): Whether to output grayscale image (default: False).

Returns

• —processed_ela_image (PIL Image): The processed ELA image.

5. gradient_processing

Description

Applies gradient filters to an image to enhance edges and transitions, which can reveal inconsistencies due to manipulation.

API Name

• —tool_gradient_processing

Parameters

• —image (PIL Image): The input image to analyze.
• —intensity (int): Intensity of the gradient effect (0-100, default: 90).
• —blue_mode (str): Mode for the blue channel. Possible values: "Abs", "None", "Flat", "Norm" (default: "Abs").
• —invert (bool): Whether to invert the gradients (default: False).
• —equalize (bool): Whether to equalize the histogram (default: False).

Returns

• —gradient_image (PIL Image): The image with gradient processing applied.

6. minmax_process

Description

Analyzes local pixel value deviations to detect subtle changes in image data, often indicative of digital forgeries.

API Name

• —tool_minmax_processing

Parameters

• —image (PIL Image): The input image to analyze.
• —channel (int): The color channel to process. Possible values: 0 (Grayscale), 1 (Blue), 2 (Green), 3 (Red), 4 (RGB Norm) (default: 4).
• —radius (int): The radius for local pixel analysis (0-10, default: 2).

Returns

• —minmax_image (PIL Image): The image with minmax processing applied.

7. augment_image_interface

Description

Applies various augmentation techniques to an image.

API Name

• —augment_image

Parameters

• —img (PIL Image): The input image to augment.
• —augment_methods (list of str): A list of augmentation methods to apply. Possible values: "rotate", "add_noise", "sharpen".
• —rotate_degrees (float): The degrees to rotate the image (0-360).
• —noise_level (float): The level of noise to add (0-100).
• —sharpen_strength (float): The strength of the sharpening effect (0-200).

Returns

• —augmented_img (PIL Image): The augmented image.

8. community_forensics_preview

Description

Provides a quick and simple prediction using our strongest model.

API Name

• —quick_predict

Parameters

• —img (str): The input image to analyze, provided as a file path.

Returns

• —(HTML): An HTML output from the loaded Gradio Space.

Behind the Scenes: Image Prediction Flow

When you upload an image for analysis and click the "Predict" button, the following steps occur:

1. Image Pre-processing and Agent Initialization

• —Image Conversion: The input image is first ensured to be a PIL (Pillow) Image object. If it's a file path, it's loaded and converted to PIL. If it's a NumPy array, it's converted. The image is then ensured to be in RGB format.
• —Agent Setup: Several intelligent agents are initialized to assist in the process:
• —EnsembleMonitorAgent: Monitors the performance of individual models.
• —ModelWeightManager: Manages and adjusts the weights of different models.
• —WeightOptimizationAgent: Optimizes model weights based on performance.
• —SystemHealthAgent: Monitors the system's resource usage (e.g., memory, GPU).
• —ContextualIntelligenceAgent: Infers context tags from the image to aid in weight adjustment.
• —ForensicAnomalyDetectionAgent: Analyzes forensic outputs for signs of manipulation.
• —System Health Monitoring: The SystemHealthAgent performs an initial check of system resources.
• —Image Augmentation (Optional): If rotate_degrees, noise_level, or sharpen_strength are provided, the image is augmented accordingly using "rotate", "add_noise", and "sharpen" methods internally. Otherwise, the original image is used.

2. Initial Model Predictions

• —Individual Model Inference: The augmented (or original) image is passed through each of the registered deepfake detection models (model_1 through model_7).
• —Performance Monitoring: For each model, the EnsembleMonitorAgent tracks its prediction label, confidence score, and inference time.
• —Result Collection: The raw prediction results (AI Score, Real Score, predicted Label) from each model are stored.

3. Smart Agent Processing and Weighted Consensus

• —Contextual Intelligence: The ContextualIntelligenceAgent analyzes the image's metadata (width, height, mode) and the raw model predictions to infer relevant context tags (e.g., "generated by Midjourney", "likely real photo"). This helps in making more informed decisions about model reliability.
• —Dynamic Weight Adjustment: The ModelWeightManager adjusts the influence (weights) of each individual model's prediction. This adjustment takes into account the initial model predictions, their confidence scores, and the detected context tags. Note that simple_prediction (Community Forensics model) is given a significantly higher base weight.
• —Weighted Consensus Calculation: A final prediction label ("AI", "REAL", or "UNCERTAIN") is determined by combining the individual model predictions using their adjusted weights. Models with higher confidence and relevance to the detected context contribute more to the final decision.
• —Performance Analysis (for Optimization): The WeightOptimizationAgent analyzes the final consensus label to continually improve the weight adjustment strategy for future predictions.

4. Forensic Processing

• —Multiple Forensic Techniques: The original image is subjected to various forensic analysis techniques to reveal hidden artifacts that might indicate manipulation:
• —Gradient Processing: Highlights edges and transitions in the image.
• —MinMax Processing: Reveals deviations in local pixel values.
• —ELA (Error Level Analysis): Performed in multiple passes (grayscale and color, with varying contrast) to detect areas of different compression levels, which can suggest tampering.
• —Bit Plane Extraction: Extracts and visualizes individual bit planes.
• —Wavelet-Based Noise Analysis: Analyzes noise patterns using wavelet decomposition.
• —Forensic Anomaly Detection: The ForensicAnomalyDetectionAgent analyzes the outputs of these forensic tools and their descriptions to identify potential anomalies or inconsistencies that could indicate image manipulation.

5. Data Logging and Output Generation

• —Inference Data Logging: All relevant data from the current prediction, including original image, inference parameters, individual model predictions, ensemble output, forensic images, and agent monitoring data, is logged to a Hugging Face dataset for continuous improvement and analysis.
• —Output Preparation: The results are formatted for display in the Gradio interface:
• —The processed image (augmented or original) is prepared.
• —The forensic analysis images are collected for display in a gallery.
• —A table summarizing each model's prediction (Model, Contributor, AI Score, Real Score, Label) is generated.
• —The raw JSON output of model results is prepared for debugging.
• —The final consensus label is prepared with appropriate styling.
• —Data Type Conversion: Numerical values (like AI Score, Real Score) are converted to standard Python floats to ensure proper JSON serialization.

Flow-Chart

<img src="graph_alt.svg">

Roadmap & Features

In Progress & Pending Tasks

Legend

• —Priority: High (Critical), Medium (Important), Low (Nice to have)
• —Status: Pending, 🔷 In Progress, ✅ Completed, 🔻 Blocked

Digital Forensics Implementation

Here's the updated table with an additional column providing instructions on how to use these tools with vision LLMs (e.g., CLIP, Vision Transformers, or CNNs) for effective AI content detection:

Top 20 Tools for AI Content Detection (with Vision LLM Integration Guidance)

Legend

• —Priority: High (Critical), Medium (Important), Low (Nice to have)
• —Status: 🔷 In-Progress, ✅ Completed, 🔻 Blocked

Hybrid Input Table for AI Content Detection (Planned)

Key Integration Tips for Hybrid Inputs

1. 1.Multimodal Models
2. 2.Use models like CLIP, BLIP, or MBT to align metadata (text) with visual features (images).
3. 3.For example: Combine a ViT (for image processing) with a Transformer (for Exif metadata or histograms).

1. 1.Feature Fusion Techniques
2. 2.Early fusion: Concatenate inputs (e.g., ELA + RGB) before the first layer.
3. 3.Late fusion: Process inputs separately and merge features before final classification.
4. 4.Cross-modal attention: Use cross-attention to align metadata with visual features (e.g., Exif text and PRNU noise maps).

1. 1.Preprocessing for Hybrid Inputs
2. 2.Normalize metadata and image data to the same scale (e.g., 0–1).
3. 3.Convert 1D histogram data into 2D images (e.g., heatmap-like plots) for consistent input formats.

1. 1.Loss Functions for Hybrid Tasks
2. 2.Use multi-task loss (e.g., classification + regression) if metadata is involved.
3. 3.For consistency checks (e.g., metadata vs. visual content), use triplet loss or contrastive loss.

Overview of Multi-Model Consensus Methods in ML