MTSAIR/MWS-Antifraud-Bench

MWS Antifraud Bench (Validation)

Experimental document-authenticity task for general-purpose multimodal language models. This is the public validation part of MWS Vision Bench anti-fraud v0.1.

The dataset is released for research and model comparison. It is not a certification tool, a production fraud-detection system, or a universal leaderboard that is expected to be resistant to deliberate optimization.

Data

The validation split contains 209 items:

• —44 ai_gen;
• —65 edited;
• —100 original.

original means an unmodified source image in this experiment. It must not be interpreted as a legal authenticity judgment.

Each item contains:

• —id;
• —type;
• —dataset_name — one of original, edited, ai_gen;
• —question;
• —answers — annotated edited fields when available;
• —image.

Language configs

The same images are available with three question-language configs:

• —default — Russian;
• —en — English;
• —zh — Chinese.

from datasets import load_dataset

antifraud_ru = load_dataset("MTSAIR/MWS-Antifraud-Bench")
antifraud_en = load_dataset("MTSAIR/MWS-Antifraud-Bench", "en")
antifraud_zh = load_dataset("MTSAIR/MWS-Antifraud-Bench", "zh")

Expected answer

The model must return one JSON object:

{"label": "original|edited|ai_gen", "arguments": "short explanation"}

Metric

The metric combines balanced three-class accuracy and the quality of the explanation for manually edited documents:

AF = 0.75 × max(0, balanced_accuracy − 1/3)
     + 0.5 × edited_reason_score

AF is aggregated over the complete split. Balanced accuracy requires the full three-class confusion matrix; unlike the five original MWS Vision Bench metrics, it is not a plain mean of independent per-item scores.

Anti-fraud is reported as a separate leaderboard category and is excluded from the primary MWS Vision Bench Overall score.

Leaderboard (Validation)

The table mirrors the Anti-fraud column in the canonical MWS Vision Bench leaderboard and is sorted by the anti-fraud score. This score is reported separately and is not included in Overall.

Limitations

This is an experimental research dataset. Its class balance and document-type distribution differ from a production stream and can provide shortcuts to a system tuned specifically for this collection. A motivated participant can optimize against the benchmark. Use the score to explore and compare models for your own scenario, not as a universal measure of document authenticity.

License and allowed use

This dataset is distributed under the MWS Antifraud Research-Only License 1.0. The dataset is intended for research and model comparison and is not a certification or production fraud-detection system.

The paragraph above is a summary. If it conflicts with LICENSE.md, the license text controls.