datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mmluMMLU (hendrycks_test on huggingface) without auxiliary train. It is much lighter (7MB vs 162MB) and faster than the original implementation, in which auxiliary train is loaded (+ duplicated!) by default for all the configs in the original version, making it quite heavy.
We use this version in tasksource.
Reference to original dataset:
Measuring Massive Multitask Language Understanding - https://github.com/hendrycks/test
@article{hendryckstest2021,
title={Measuring Massive Multitask Language… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/mmlu.bigbenchBIG-Bench but it doesn't require the hellish dependencies (tensorflow, pypi-bigbench, protobuf) of the official version.
dataset = load_dataset("tasksource/bigbench",'movie_recommendation')
Code to reproduce:
https://colab.research.google.com/drive/1MKdLdF7oqrSQCeavAcsEnPdI85kD0LzU?usp=sharing
Datasets are capped to 50k examples to keep things light.
I also removed the default split when train was available also to save space, as default=train+val.
@article{srivastava2022beyond… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/bigbench.proofwriter
Dataset Card for "proofwriter"
More Information needed
lsat-lr
Dataset Card for "lsat-lr"
More Information needed
babi_nli
bAbi_nli
bAbI tasks recasted as natural language inference.
https://github.com/facebookarchive/bAbI-tasks
tasksource recasting code:
https://colab.research.google.com/drive/1J_RqDSw9iPxJSBvCJu-VRbjXnrEjKVvr?usp=sharing
@article{weston2015towards,
title={Towards ai-complete question answering: A set of prerequisite toy tasks},
author={Weston, Jason and Bordes, Antoine and Chopra, Sumit and Rush, Alexander M and Van Merri{\"e}nboer, Bart and Joulin, Armand and Mikolov, Tomas}… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/babi_nli.lsat-rc
Dataset Card for "lsat-rc"
More Information needed
lsat-ar
Dataset Card for "lsat-ar"
More Information needed
ScienceQA_text_only
Dataset Card for "scienceQA_text_only"
ScienceQA text-only examples (examples where no image was initially present, which means they should be doable with text-only models.)
@article{10.1007/s00799-022-00329-y,
author = {Saikh, Tanik and Ghosal, Tirthankar and Mittal, Amish and Ekbal, Asif and Bhattacharyya, Pushpak},
title = {ScienceQA: A Novel Resource for Question Answering on Scholarly Articles},
year = {2022},
journal = {Int. J. Digit. Libr.},
month = {sep}
}
brainteasers
Dataset Card for "brainteasers"
More Information needed
esci
Dataset Card for "esci"
ESCI product search dataset
https://github.com/amazon-science/esci-data/
Preprocessings:
-joined the two relevant files
-product_text aggregate all product text
-mapped esci_label to full name
@article{reddy2022shopping,
title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
author={Chandan K. Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/esci.tasksource-instruct
Dataset Card for "tasksource-instruct-v0" (TSI)
Multi-task instruction-tuning data recasted from 485 of the tasksource datasets.
Dataset size is capped at 30k examples per task to foster task diversity.
!pip install tasksource, pandit
import tasksource, pandit
df = tasksource.list_tasks(instruct=True).sieve(id=lambda x: 'mmlu' not in x)
for tasks in df.id:
yield tasksource.load_task(task,instruct=True,max_rows=30_000,max_rows_eval=200)
https://github.com/sileod/tasksource… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/tasksource-instruct.ruletaker
Dataset Card for "ruletaker"
https://github.com/allenai/ruletaker
@inproceedings{ruletaker2020,
title = {Transformers as Soft Reasoners over Language},
author = {Clark, Peter and Tafjord, Oyvind and Richardson, Kyle},
booktitle = {Proceedings of the Twenty-Ninth International Joint Conference on
Artificial Intelligence, {IJCAI-20}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Christian Bessiere}… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/ruletaker.defeasible-nlihttps://github.com/rudinger/defeasible-nli
@inproceedings{rudinger2020thinking,
title={Thinking like a skeptic:
feasible inference in natural language},
author={Rudinger, Rachel and Shwartz, Vered and Hwang, Jena D and Bhagavatula, Chandra and Forbes, Maxwell and Le Bras, Ronan and Smith, Noah A and Choi, Yejin},
booktitle={Findings of the Association for Computational Linguistics: EMNLP 2020},
pages={4661--4675},
year={2020}
}
LogicNLI
Dataset Card for "LogicNLI"
@inproceedings{tian-etal-2021-diagnosing,
title = "Diagnosing the First-Order Logical Reasoning Ability Through {L}ogic{NLI}",
author = "Tian, Jidong and
Li, Yitian and
Chen, Wenqing and
Xiao, Liqiang and
He, Hao and
Jin, Yaohui",
editor = "Moens, Marie-Francine and
Huang, Xuanjing and
Specia, Lucia and
Yih, Scott Wen-tau",
booktitle = "Proceedings of the 2021 Conference on Empirical… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/LogicNLI.logical-fallacyhttps://github.com/causalNLP/logical-fallacy
@article{jin2022logical,
title={Logical fallacy detection},
author={Jin, Zhijing and Lalwani, Abhinav and Vaidhya, Tejas and Shen, Xiaoyu and Ding, Yiwen and Lyu, Zhiheng and Sachan, Mrinmaya and Mihalcea, Rada and Sch{\"o}lkopf, Bernhard},
journal={arXiv preprint arXiv:2202.13758},
year={2022}
}
logical-entailmenthttps://github.com/google-deepmind/logical-entailment-dataset
@inproceedings{
evans2018can,
title={Can Neural Networks Understand Logical Entailment?},
author={Richard Evans and David Saxton and David Amos and Pushmeet Kohli and Edward Grefenstette},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=SkZxCk-0Z},
}
ecqa
Dataset Card for "ecqa"
https://github.com/dair-iitd/ECQA-Dataset
@inproceedings{aggarwaletal2021ecqa,
title={{E}xplanations for {C}ommonsense{QA}: {N}ew {D}ataset and {M}odels},
author={Shourya Aggarwal and Divyanshu Mandowara and Vishwajeet Agrawal and Dinesh Khandelwal and Parag Singla and Dinesh Garg},
booktitle="Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/ecqa.tasksource-jev-typed-decisions
tasksource-jev-typed-decisions
2.5 million typed decisions (choices, ratings and probabilities) from 670 sources.
Why use it
Real supervision. Labels, ratings, and annotator votes come from
established datasets, not a teacher model. Every row names its source.
Breadth. Over 300 dataset families: NLI and reasoning, QA and
commonsense, sentiment, intent and topic, toxicity and safety, preference
pairs, fact checking, entity tagging, and dozens of languages. GLUE… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions.QuALITY
Dataset Card for "QuALITY"
@article{bowman2022quality,
title={QuALITY: Question Answering with Long Input Texts, Yes!},
author={Bowman, Samuel R and Chen, Angelica and He, He and Joshi, Nitish and Ma, Johnny and Nangia, Nikita and Padmakumar, Vishakh and Pang, Richard Yuanzhe and Parrish, Alicia and Phang, Jason and others},
journal={NAACL 2022},
year={2022}
}
lingnlihttps://github.com/Alicia-Parrish/ling_in_loop/
@inproceedings{parrish-etal-2021-putting-linguist,
title = "Does Putting a Linguist in the Loop Improve {NLU} Data Collection?",
author = "Parrish, Alicia and
Huang, William and
Agha, Omar and
Lee, Soo-Hwan and
Nangia, Nikita and
Warstadt, Alexia and
Aggarwal, Karmanya and
Allaway, Emily and
Linzen, Tal and
Bowman, Samuel R.",
booktitle = "Findings of the Association for… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/lingnli.Boardgame-QAhttps://arxiv.org/pdf/2306.07934.pdf
chaos-mnli-ambiguity
chaos-mnli-ambiguity
ChaosNLI, MNLI portion: 1,599 MNLI pairs relabeled by 100 annotators each (Nie et al., 2020).
label_dist and label_count follow the entailment/neutral/contradiction order, and gini is the Gini
coefficient of label_dist (0 = annotators evenly split, 1 = unanimous). Built from the jsonl first uploaded
here, which flattens the ChaosNLI release (https://github.com/easonnie/ChaosNLI) and adds gini; the
variable-key label_counter (a duplicate of label_count) is… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/chaos-mnli-ambiguity.robustLR
Dataset Card for "robustLR"
https://github.com/INK-USC/RobustLR
@article{sanyal2022robustlr,
title={Robustlr: Evaluating robustness to logical perturbation in deductive reasoning},
author={Sanyal, Soumya and Liao, Zeyi and Ren, Xiang},
journal={arXiv preprint arXiv:2205.12598},
year={2022}
}
doc-nli
Dataset Card for "doc-nli"
https://github.com/salesforce/DocNLI/tree/main
@inproceedings{yin-etal-2021-docnli,
title = "{D}oc{NLI}: A Large-scale Dataset for Document-level Natural Language Inference",
author = "Yin, Wenpeng and
Radev, Dragomir and
Xiong, Caiming",
editor = "Zong, Chengqing and
Xia, Fei and
Li, Wenjie and
Navigli, Roberto",
booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/doc-nli.planbench
Dataset Card for "planbench"
https://arxiv.org/abs/2206.10498
@article{valmeekam2024planbench,
title={Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change},
author={Valmeekam, Karthik and Marquez, Matthew and Olmo, Alberto and Sreedharan, Sarath and Kambhampati, Subbarao},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
cladderhttps://github.com/causalNLP/cladder
patent-phrase-similarityFOL-nli
Dataset Card for "FOL-nli"
https://github.com/sileod/unigram/
https://arxiv.org/abs/2406.11035
Citation:
@article{sileo2024scaling,
title={Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars},
author={Sileo, Damien},
journal={arXiv preprint arXiv:2406.11035},
year={2024}
}
AbductionRules
Dataset Card for "AbductionRules"
More Information needed
clutrr
clutrr
CLUTRR v1 (gen_train234_test2to10): infer a kinship relation from a short story.
Original data: CLUTRR/v1. Repackaged as parquet for tasksource by scripts/upload_repackaged.py.