jordansp/multimodal-grounding-ooc

Multimodal Grounding of Explanations for Out-of-Context Misinformation Detection

This dataset contains the outputs, explanations, and visual grounding audits for three vision-language model configurations evaluated on out-of-context (OOC) misinformation detection:

1. 1.Gemma-4-31B-It (Direct): Baseline API evaluation with minimal thinking compute.
2. 2.Gemma-4-31B-It (Thinking): Deliberation API evaluation with high thinking compute (up to 4,352 tokens).
3. 3.Gemma-3-27B-It (Direct): Prior-generation VLM baseline (does not support thinking).

The dataset is designed to support research in explainable fact-checking, visual faithfulness, hallucination detection, and inference-time compute scaling in multimodal models.

Dataset Access & Images

Due to licensing and copyright terms, we do not distribute the raw image files in this repository. To construct the complete multimodal inputs, you must download the images directly from the original benchmark datasets and match them using the sample_id field:

• —COSMOS Images: Available via request from the COSMOS GitHub Repository.
• —NewsCLIPpings Images: Available from the NewsCLIPpings GitHub Repository (requires setting up the source VisualNews dataset).

Dataset Structure

The packaged data is divided into three CSV files located under the data/ directory:

1. Main Evaluation (data/main_evaluation.csv)

This file contains the final classification predictions, generated explanations, thought traces (where applicable), and automated Gemini-3.1-Flash-Lite judge scores for all evaluated samples under the primary prompt (prompt_v5).

• —Rows: 8,969 records (representing evaluation samples across all three models).
• —Key Fields:
• —sample_id: Unique identifier (matching COSMOS or NewsCLIPpings test sets).
• —model_key: VLM configuration name (gemma-4-31b-it-direct, gemma-4-31b-it-thinking, or gemma-3-27b-it-direct).
• —dataset: Source benchmark (cosmos or newsclippings).
• —tested_caption: The textual caption supplied to the model.
• —ground_truth_label: True relationship (In-Context or Out-of-Context).
• —predicted_confidence: Model's Out-of-Context confidence score (0–100).
• —predicted_label: Predicted relationship based on the 50-confidence threshold.
• —explanation: The natural-language rationale generated by the VLM.
• —thinking_trace: The raw thought trace (Gemma-4 Thinking only).
• —thinking_token_count: Number of thinking tokens generated.
• —latency_seconds: Response latency in seconds.
• —judge_grounding_score: Automated visual grounding score (1–5 scale).
• —is_grounded: Boolean indicating if the score meets the grounding threshold ($\geq 4$).
• —hallucination_present: Boolean flag indicating if the model's explanation contains claims about the image that are completely fabricated, hallucinated, or directly contradicted by the visual content (e.g., claiming a person is wearing a specific color that they are not, or claiming an object is present when it is absent).
• —supported_visual_claims: Semicolon-separated list of assertions made in the explanation that are verified by the judge as true and directly observable in the image (e.g., "microphone in front of speaker", "red and yellow clothing").
• —unsupported_visual_claims: Semicolon-separated list of assertions in the explanation that are refuted or entirely unsupported by the image (e.g., claiming a flag has a specific symbol that is not visible).
• —not_visually_checkable_claims: Semicolon-separated list of assertions in the explanation that require external/historical knowledge and cannot be verified from the image pixels alone (e.g., specific names, dates, or off-camera events like "taken in Vladivostok" or "the man is former HUD secretary Julian Castro").
• —judge_feedback: Explanatory rationale provided by the Gemini auditor explaining the grounding score.

2. Perturbation Tests (data/perturbation_tests.csv)

Contains results for perturbation experiments designed to test model robustness and warranted vs. unwarranted sensitivity.

• —Rows: 20,479 records.
• —Perturbation Types:
• —caption_swap: OOC-origin samples evaluated under the corrected caption. The corrected-caption IC rate uses all valid swap outputs; the baseline-conditioned reclassification rate uses only rows whose baseline_prediction is Out-of-Context and measures how often perturbed_prediction becomes In-Context.
• —bias_false_authority: Captions prepended with an authoritative false cue (e.g., "A reverse image search reveals...").
• —bias_fact_checker_verified: Captions prepended with an authoritative true cue (e.g., "According to fact-checkers...").
• —Key Fields:
• —sample_id, model_key, dataset, original_caption, tested_caption.
• —perturbation_type: caption_swap, bias_false_authority, or bias_fact_checker_verified.
• —true_label: True relationship.
• —baseline_prediction: Actual prediction under the primary unperturbed prompt. It is not necessarily Out-of-Context for a caption-swap row.
• —perturbed_confidence: Model's OOC confidence rating under the perturbation.
• —perturbed_prediction: Classification under the perturbation.
• —explanation, thinking_trace.

3. Human Calibration (data/human_calibration.csv)

A stratified subset of 150 explanations independently audited by a human expert using the same five-point visual grounding rubric, allowing researchers to study human-AI alignment on visual explanation quality.

• —Rows: 150 records.
• —Key Fields:
• —sample_id, model_key, dataset, tested_caption.
• —human_score: Grounding score assigned by the human rater (1–5).
• —human_notes: Human comments and qualitative rationale.
• —automated_judge_score: Grounding score assigned by Gemini-3.1-Flash-Lite.
• —automated_judge_feedback: Gemini judge's justification.

Evaluation Prompts

VLM Inference Prompt (prompt_v5.md)

The models were evaluated using the following instructions:

System Prompt:

You are an expert fact-checker evaluating whether a news image and its caption describe the same real-world event. Approximately half of the pairs you will see are genuine matches and half are mismatches — do not assume either direction.

User Prompt:

Caption to evaluate: "{{caption}}"

Instructions:
1. Visual Analysis: Examine the image carefully. Identify specific visual evidence — visible text, faces, objects, clothing, settings, landmarks, or other identifiable details.
2. Contradiction Check: Look for direct visual evidence that *contradicts* a specific claim in the caption. A mismatch requires a positive reason to doubt — not merely an inability to confirm every detail. Unverifiable specifics (exact dates, off-screen locations, named individuals without visible identification) are not evidence of mismatch.
3. Respond in JSON with exactly two fields:
   - "confidence": an integer 0-100 representing your confidence that this pair is Out-of-Context, where 0 = certainly In-Context and 100 = certainly Out-of-Context. Use the full range — values like 25, 35, 65, or 80 are expected and encouraged. Only use 0, 50, or 100 when you are completely certain or completely uncertain.
   - "explanation": a concise 2-3 sentence justification citing the specific visual evidence that drove your decision. Begin directly with your analysis — no preamble.

Judge Prompt (judge_prompt.md)

The rubric and instructions given to the Gemini-3.1-Flash-Lite auditor that produced the grounding scores in main_evaluation.csv.

Prompt Development History (prompt_versions/)

Every result in this dataset was produced under v5. The four earlier versions are included so the development history stated in the dissertation can be inspected rather than taken on trust.

Each version was trialled on evaluation samples and revised by reading the resulting outputs, not by comparing accuracy between versions, so the changes below responded to observed failure modes rather than to measured performance. The trials nonetheless used evaluation samples rather than a separate development split, so prompt design was not blind to the test data and absolute accuracy figures may be mildly optimistic. The matched Direct/Thinking comparison is unaffected, because all three model conditions use v5. Outputs from the earlier versions are not released; only the prompt texts are.

The v2 → v3 correction matters for interpreting the results: v5 asks for contradiction rather than confirmation, which is why an In-Context verdict can rest on the absence of a visible contradiction. The dissertation records the resulting tension with the judge rubric, which rewards affirmative visual support, as a limitation.

Citation

If you use this dataset or findings in your work, please cite the dissertation:

@thesis{speight2026narrowing,
  author    = {Speight, Jordan},
  title     = {Narrowing the Grounding Gap: Evaluating Inference-Time Reasoning in Vision-Language Models for Out-of-Context Misinformation Detection},
  year      = {2026},
  school    = {University of Essex},
  type      = {Master's thesis}
}