datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
bigbenchBIG-Bench but it doesn't require the hellish dependencies (tensorflow, pypi-bigbench, protobuf) of the official version.
dataset = load_dataset("tasksource/bigbench",'movie_recommendation')
Code to reproduce:
https://colab.research.google.com/drive/1MKdLdF7oqrSQCeavAcsEnPdI85kD0LzU?usp=sharing
Datasets are capped to 50k examples to keep things light.
I also removed the default split when train was available also to save space, as default=train+val.
@article{srivastava2022beyond… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/bigbench.tasksource-instruct
Dataset Card for "tasksource-instruct-v0" (TSI)
Multi-task instruction-tuning data recasted from 485 of the tasksource datasets.
Dataset size is capped at 30k examples per task to foster task diversity.
!pip install tasksource, pandit
import tasksource, pandit
df = tasksource.list_tasks(instruct=True).sieve(id=lambda x: 'mmlu' not in x)
for tasks in df.id:
yield tasksource.load_task(task,instruct=True,max_rows=30_000,max_rows_eval=200)
https://github.com/sileod/tasksource… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/tasksource-instruct.FOL-nli
Dataset Card for "FOL-nli"
https://github.com/sileod/unigram/
https://arxiv.org/abs/2406.11035
Citation:
@article{sileo2024scaling,
title={Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars},
author={Sileo, Damien},
journal={arXiv preprint arXiv:2406.11035},
year={2024}
}
prm800k_dpo
PRM800K preference pairs (prompt / chosen / rejected)
PRM800K (OpenAI, Let's Verify Step by Step) turned into clean
preference pairs, ready for DPO / reward modeling (TRL "standard" preference format).
Raw source: tasksource/PRM800K.
Train/test follow the original MATH benchmark splits (see Splits below).
from datasets import load_dataset
ds = load_dataset("tasksource/prm800k_dpo", "solution") # or "step"
Configs
solution — whole-solution preference… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/prm800k_dpo.icl-symbol-tuning-instruct
Description
Few-shot prompting demonstrates that language models can learn in context even though they were not trained to do. However, explicitly learning to learn in context meta-icl leads to better results. With symbol tuning, labels are replaced with arbitrary symbols (e.g. foo/bar), which makes learning in context a key condition to learn the instructions
We implement symbol tuning, as presented in the Symbol tuning improves in-context learning paper with tasksource… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/icl-symbol-tuning-instruct.tasksource_dpo_pairs
Dataset Card for "tasksource_dpo_pairs"
The tasksource collection as pairs for DPO or RLHF.
The tasksource does not contains LLM-generated data. It gathers many up to dates expert-constructed datasets, notably on NLI and logical reasoning.
https://aclanthology.org/2024.lrec-main.1361.pdf
@inproceedings{sileo-2024-tasksource-large,
title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
author = "Sileo, Damien",
editor =… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/tasksource_dpo_pairs.