factuality
self-alignment-for-factualityThe data was organized and utilized in Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation.
If you find our data useful, please cite our work using the following reference:
@inproceedings{zhang-etal-2024-self,
title = "Self-Alignment for Factuality: Mitigating Hallucinations in {LLM}s via Self-Evaluation",
author = "Zhang, Xiaoying and
Peng, Baolin and
Tian, Ye and
Zhou, Jingyan and
Jin, Lifeng and
Song, Linfeng and… See the full description on the dataset page: https://huggingface.co/datasets/xyingzhang/self-alignment-for-factuality.xsum_factualityNeural abstractive summarization models are highly prone to hallucinate content that is unfaithful to the input
document. The popular metric such as ROUGE fails to show the severity of the problem. The dataset consists of
faithfulness and factuality annotations of abstractive summaries for the XSum dataset. We have crowdsourced 3 judgements
for each of 500 x 5 document-system pairs. This will be a valuable resource to the abstractive summarization community.news_media_bias_and_factuality
News Media Factual Reporting and Political Bias
Dataset introduced in the paper "Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions" published in the CLEF 2024 main conference.
Similar to the news media reliability dataset, this dataset consists of a collections of 4K new media domains names with political bias and factual reporting labels.
Columns of the dataset:
source: domain name
bias: the political bias label. Values: "left"… See the full description on the dataset page: https://huggingface.co/datasets/sergioburdisso/news_media_bias_and_factuality.event_factuality
Event Factuality (It Happened / UDS-IH2)
Source
Decomp “It Happened” (UDS-IH2): https://decomp.io/projects/factuality/
UD English-EWT v1.2 (r1.2) for sentence reconstruction: https://github.com/UniversalDependencies/UD_English-EWT/tree/r1.2
Contains raw web/news text; included for research purposes only (no endorsement).
Task
Binary predicate-level event factuality (one row per predicate).
Labels
label: 0=false, 1=true
label_rule: single, agree, na_other, tie_conf4_vs0… See the full description on the dataset page: https://huggingface.co/datasets/compling/event_factuality.OpenDataGen-factuality-en-v0.1This synthetic dataset was generated using the Open DataGen Python library. (https://github.com/thoddnn/open-datagen)
Methodology:
Retrieve random article content from the HuggingFace Wikipedia English dataset.
Construct a Chain of Thought (CoT) to generate a Multiple Choice Question (MCQ).
Utilize a Large Language Model (LLM) to score the results then filter it.
All these steps are prompted in the 'template.json' file located in the specified code folder.
Code:… See the full description on the dataset page: https://huggingface.co/datasets/thoddnn/OpenDataGen-factuality-en-v0.1.factuality-rmbench-style
Factuality RM-Bench Style
Factuality RM-Bench Style is a controlled English dataset for studying whether
reward models and representation probes prefer stylistic presentation over
factual correctness. Each row contains one question, a localized correct and
incorrect proposition, and six responses formed by crossing correctness with
three presentation styles: concise, normal, and Markdown.
This repository is an export package for
factuality_rmbench_style_v6. The published data… See the full description on the dataset page: https://huggingface.co/datasets/Yunnnuy/factuality-rmbench-style.