datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
llm-disagreement
Lenz Frontier-LLM Disagreement — v1.1
Five frontier language models each rated the same 997 real fact-check claims submitted by users of Lenz. This dataset is the per-claim record of where they agreed and where they did not.
In 63% of real-world fact-checks, top AI models don't agree on the answer — at least one model dissents from the majority, or no majority forms at all (95% CI 60–66%).
At a glance
Claims
997 complete (of 1,000 harvested)
Models… See the full description on the dataset page: https://huggingface.co/datasets/DavidYor06/llm-disagreement.SBIC_DisagreementThis dataset is processed version of Social Bias Inference Corpus(SBIC) dataset including text, annotator's demographics and the annotation disagreement labels.
Paper: Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Authors: Ruyuan Wan, Jaehyung Kim, Dongyeop Kang
Github repo: https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement
Dynasent_DisagreementThis dataset is processed version of Dynamic Sentiment Analysis (DynaSent) dataset including text and the annotation disagreement labels.
Paper: Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Authors: Ruyuan Wan, Jaehyung Kim, Dongyeop Kang
Github repo: https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement
Source Data: Dynamic Sentiment Analysis Dataset(Potts et al. 2021)
SChem_DisagreementThis dataset is processed version of Social Chemistry 101(SChem) dataset including text and the annotation disagreement labels.
Paper: Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Authors: Ruyuan Wan, Jaehyung Kim, Dongyeop Kang
Github repo: https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement
Source Data: Social Chemistry 101(Forbes et al. 2020)
Politeness_DisagreementThis dataset is processed version of Stanford Politeness Corpus (Wikipedia) including text and the annotation disagreement labels.
Paper: Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Authors: Ruyuan Wan, Jaehyung Kim, Dongyeop Kang
Github repo: https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement
Source Data: Wikipedia Politeness Corpus(Danescu-Niculescu-Mizil et al. 2013)
Dilemmas_DisagreementThis dataset is processed version of Dilemmas dataset including text and the annotation disagreement labels.
Paper: Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Authors: Ruyuan Wan, Jaehyung Kim, Dongyeop Kang
Github repo: https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement
Source Data: Scruples-dilemmas (Lourie, Bras, and Choi 2021)