teplitsa-soc-tech/factbutcher-benchmark

English · Русский

FactButcher Russian Fact-Checking Dataset

This dataset contains 423 claims in Russian and the results of checking them. Some claims are true, some are false, and some allow more than one defensible answer.

You can see what kinds of claims people bring to fact-checkers, investigate a few of them yourself, or use the complete collection to compare different fact-checking tools.

A few examples

These are actual rows from the dataset. In the data files, the statement is stored in claim and its main reviewed result is stored in gold_verdict.

View all claims

Open the `CSV file` in your browser, or download it and use Excel, Google Sheets, LibreOffice, or another spreadsheet program. Each row contains one claim and the result of checking it.

If you want to investigate a claim before seeing the answer, hide the gold_verdict and acceptable_verdicts columns. Provereno.Media rows include a published fact-check in source_url. For claims that depend on time, reference_date shows the date against which the verdict was assigned.

Where the data came from

The dataset has two parts:

An LLM was used to extract factual claims from the FactButcher requests. The claims were then edited into short, self-contained statements, checked, and reviewed by a person. The published statements are not verbatim user messages; the original messages are not included in the dataset.

The Provereno.Media rows were adapted from published articles. Each row links to the original fact-check. The 149 rows point to 146 pages because two articles contribute more than one separate claim.

The collection, checking, and labeling process is described in `METHODOLOGY.md`.

Test a model or service

You can use the dataset as a shared collection of questions with reviewed answers. To do this:

1. 1.send each claim to the model, service, or fact-checking script you want to test;
2. 2.save its verdict together with the corresponding claim_id;
3. 3.map the answers to TRUE, FALSE, MIXED, or INSUFFICIENT_EVIDENCE;
4. 4.compare them with the reference answers in the dataset.

How claims are sent depends on the tool you choose. An API-based tool will require its own small connecting script and access settings. This repository contains the claims and reference answers.

For a comparable result, run all 423 rows under the same conditions and report the model, prompt, and web-search settings. Also state whether provereno.media was accessible: a tool with web search may find the published fact-check there.

The dataset has one split, test. Do not use these rows to train or tune a system and then publish its result as an independent evaluation.

How the verdicts work

gold_verdict is the main reviewed result:

• —TRUE — the claim is supported;
• —FALSE — the claim is contradicted;
• —MIXED — important parts of the claim have different truth values, or reliable sources do not support one unambiguous answer.

In some cases, two neighboring verdicts are defensible. They are listed in acceptable_verdicts. This supports two ways of scoring results:

• —acceptable accuracy: the system's answer appears in acceptable_verdicts;
• —strict accuracy: the system's answer equals gold_verdict.

INSUFFICIENT_EVIDENCE can appear among the accepted answers, but it is not used as the main verdict.

There are 120 rows with more than one accepted verdict.

Data files

• —`data/factbutcher_benchmark_v1.csv` is intended for spreadsheets. Multiple accepted verdicts are separated by |.
• —`data/factbutcher_benchmark_v1.jsonl` is the main typed version for software. Each line contains one JSON object.

Load the JSONL file with standard Python:

import json

with open("data/factbutcher_benchmark_v1.jsonl", encoding="utf-8") as file:
    rows = [json.loads(line) for line in file]

Or use the Hugging Face datasets library:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files={"test": "data/factbutcher_benchmark_v1.jsonl"},
)

Fields

The full machine-readable field specification is in `metadata/schema.json`.

Limitations

• —This is a relatively small Russian-language collection.
• —It reflects FactButcher requests and selected Provereno.Media coverage, not every possible fact-checking topic.
• —Topics and verdicts occur naturally rather than in equal proportions.
• —Some claims depend on time.
• —Even a human-reviewed verdict can be contestable. Multiple accepted labels represent some, but not all, ambiguity.
• —Human Benchmark rows do not include a complete evidence bundle or written fact-check. Provereno.Media rows link to the published article.

Privacy

The dataset does not contain original Telegram messages, user identifiers, or private FactButcher operational data. It publishes only the prepared claims and the fields needed to use them.

License and citation

The dataset is available under Creative Commons Attribution 4.0 International.

Every Provereno.Media row links to its original article and includes source license information. See `NOTICE.md` for details. Citation metadata is available in `CITATION.cff`.

File validation

This command is intended for maintainers and people who mirror or repackage the dataset. It checks that the data, metadata, and checksums agree. You do not need to run it simply to browse the dataset.

python scripts/validate_dataset.py