datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
oasst2
Open Assistant Conversations Dataset Release 2 (OASST2)
Dataset Structure
This dataset contains message trees. Each message tree has an initial prompt message as the root node,
which can have multiple child messages as replies, and these child messages can have multiple replies.
All messages have a role property: this can either be "assistant" or "prompter". The roles in
conversation threads from prompt to leaf node strictly alternate between "prompter" and… See the full description on the dataset page: https://huggingface.co/datasets/OpenAssistant/oasst2.oasst2-33k-ja
oasst2-33k-ja
This repository provides an instruction tuning dataset developed by LLM-jp, a collaborative project launched in Japan.
The dataset comprises a Japanese translation of an English subset from oasst2, translated using DeepL.
The English subset can be found here.
For the creation of this dataset, we processed data from kunishou/oasst2-135k-ja.
Send Questions to
llm-jp(at)nii.ac.jp
Model Card Authors
The names are listed in alphabetical order.… See the full description on the dataset page: https://huggingface.co/datasets/llm-jp/oasst2-33k-ja.oasst1-oasst2-ja_single_dedupedoasst2-openai-native
oasst2-openai-native
A deterministic, native OpenAI/TRL reconstruction of
OpenAssistant/oasst2.
It turns the original flat parent_id message table into two directly usable
configs without LLM transformation: multilingual SFT conversations and ranked
DPO preference pairs.
At a glance
Config
Train
Test
Unit
sft
12,717
671
alternating conversation ending in assistant
dpo
42,639
2,284
prompt + chosen/rejected assistant pair
The data is multilingual:… See the full description on the dataset page: https://huggingface.co/datasets/Archangel-system/oasst2-openai-native.alexredna_oasst2_dpo_pairshttps://huggingface.co/datasets/alexredna/oasst2_dpo_pairs
oasst2_orpo_mix_tokenizer_phi_3_v1
https://huggingface.co/datasets/NickyNicky/orpo-dpo-mix-54k
oasst2_pairwise_rlhf_reward
Dataset Card for "oasst2_pairwise_rlhf_reward"
import pandas as pd
from datasets import load_dataset,concatenate_datasets, Dataset, DatasetDict
import numpy as np
dataset = load_dataset("OpenAssistant/oasst2")
df=concatenate_datasets(list(dataset.values())).to_pandas()
m2t=df.set_index("message_id")['text'].to_dict()
m2r=df.set_index("message_id")['role'].to_dict()
m2p=df.set_index('message_id')['parent_id'].to_dict()
m2history=dict() # message id to unrolled history
for k,v in… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/oasst2_pairwise_rlhf_reward.oasst2
OASST2 Converted to Chat Format
This dataset is a converted version of OpenAssistant/oasst2 in OpenAI chat format with smart system prompts.
System Prompts
The dataset includes smart system prompts assigned based on conversation content:
Coding: Programming, debugging, algorithms
Math/Science: Mathematics, physics, chemistry, biology
Creative Writing: Stories, poems, creative content
Academic: Research, scholarly writing, analysis
Business: Professional advice, strategy… See the full description on the dataset page: https://huggingface.co/datasets/charlie-ohxyz/oasst2.oasst2_egyptian_arabic_convsoasst2_thai_top1_chat_format
Open Assistant 2 Top-1 Thai
Dataset Details
Dataset Description
A top-1 Thai dataset taken from the top scoring https://huggingface.co/datasets/OpenAssistant/oasst2 conversations. Saved in HF Chat format.
License: Apache 2.0
Script: https://github.com/wannaphong/deep_4_all/tree/main/datasets/oasst
Dataset Structure
We structure the dataset using the format commonly used as input into Hugging Face Chat Templates:
[
{'content':… See the full description on the dataset page: https://huggingface.co/datasets/pythainlp/oasst2_thai_top1_chat_format.oasst2_dense_flat
Dataset Card for "oasst2_dense_flat"
More Information needed
oasst2-135k-jaUpdate:
2023/12/25oasst2-135k-jaをチャット形式に変換したoasst2-chat-68k-jaを公開しました。
This dataset was created by automatically translating "OpenAssistant/oasst2" into Japanese by DeepL.
"OpenAssistant/oasst2" を DeepL翻訳を用いて日本語に自動翻訳したデータセットになります。
以下のコードを用いることで、 Instruction と Output （prompterの命令とassistantの回答）の形式に変換することができます。
ファインチューニングで使用する場合はこちらのコードで変換して下さい（変換には5分程度かかります）。
変換コード参考https://github.com/h2oai/h2o-llmstudio/blob/5ebfd3879e226b4e1afd0a0b45eb632e60412129/app_utils/utils.py#L1888
pip install… See the full description on the dataset page: https://huggingface.co/datasets/kunishou/oasst2-135k-ja.oasst2_caoasst2_clusters
max count_word cluster_1: 1722
min count_word cluster_1: 11
max count_word cluster_2: 2624
min count_word cluster_2: 21
max count_word cluster_3: 2370
min count_word cluster_3: 31
DatasetDict({
Cluster_1: Dataset({
features: ['Text', 'Cluster', 'Polarity', 'count_word'],
num_rows: 4797
})
Cluster_2: Dataset({
features: ['Text', 'Cluster', 'Polarity', 'count_word'],
num_rows: 4025
})
Cluster_3: Dataset({
features: ['Text', 'Cluster'… See the full description on the dataset page: https://huggingface.co/datasets/NickyNicky/oasst2_clusters.oasst2_chatml link: https://huggingface.co/datasets/OpenAssistant/oasst2
Message counts by language:
en: 64,513
es: 28,199
ru: 13,935
zh: 8,615
de: 6,145
fr: 3,880
pt-BR: 2,699
th: 1,560
ca: 1,283
it: 943
uk-UA: 845
ja: 788
pl: 435
eo: 295
eu: 274
vi: 207
fi: 138
hu: 113
ar: 80
nl: 72
da: 44
tr: 37
ko: 24
he: 24
id: 12
cs: 12
bn: 1
sv: 1
oasst2_es_instruct_hfThis is the Spanish subset from the OpenAssistant/oasst2 dataset.
The dataset has been extracted from the 2023-11-05_oasst2_ready.trees.jsonl.gz file to parse all the conversation trees and put it in a huggingface-friendly format so you can use apply_chat_template as explained on the Chat Templating documentation.
Example
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
chat = [
{"role": "user"… See the full description on the dataset page: https://huggingface.co/datasets/bertin-project/oasst2_es_instruct_hf.oasst2_low_chs_worst_rejoasst2-chat-68k-jaoasst2-135k-jaをチャット形式に変換したデータセットになります。マルチターン会話でのファインチューニングをする際にご活用下さい（1レコードのトークン長が大きいのでそれなりの計算リソースが必要になります）。フォーマットは ShareGPT 形式になっています。ファインチューニングをする際はこちらの記事を参考にして下さい。
OpenAssistant/oasst2https://huggingface.co/datasets/OpenAssistant/oasst2
oasst2_ca_gptoasst2_top1_chat_format
OpenAssistant TOP-1 Conversation Threads in huggingface chat format
Export of oasst2 only top 1 threads in huggingface chat format
Script
The convert script can be find here
oasst2_euskera_gpt2023-11-05-oasst2-ready-trees-sft-instr
Dataset Card for "2023-11-05-oasst2-ready-trees-sft-instr"
More Information needed
ChatML-oasst2_dpo_pairs_ensablo/oasst2_dpo_pairs_en in ChatML format, ready to use in HuggingFace TRL's DPO Trainer.
Python code used for conversion:
from datasets import load_dataset
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Felladrin/Llama-160M-Chat-v1")
dataset = load_dataset("sablo/oasst2_dpo_pairs_en", split="train")
def format(columns):
return {
"prompt": tokenizer.apply_chat_template(columns["chosen"][:-1], tokenize=False, add_generation_prompt=True)… See the full description on the dataset page: https://huggingface.co/datasets/Felladrin/ChatML-oasst2_dpo_pairs_en.oasst2_en_dpoI use the following code to process the original OpenAssistant/oasst2 dataset.
oasst = dt.load_dataset("OpenAssistant/oasst2", split="train")
oasst_en = oasst.filter(lambda x: x["lang"] == "en", num_proc=8)
def convert_to_standard_dpo(flat_dataset):
"""
Convert a flat OpenAssistant dataset to a standard DPO format with columns:
- prompt: string (root prompt)
- chosen: list of dicts representing the chosen dialogue path
- rejected: list of dicts representing the rejected… See the full description on the dataset page: https://huggingface.co/datasets/BigCatc/oasst2_en_dpo.oasst2
Open Assistant Conversations Dataset Release 2 (OASST2)
Dataset Structure
This dataset contains message trees. Each message tree has an initial prompt message as the root node,
which can have multiple child messages as replies, and these child messages can have multiple replies.
All messages have a role property: this can either be "assistant" or "prompter". The roles in
conversation threads from prompt to leaf node strictly alternate between "prompter" and… See the full description on the dataset page: https://huggingface.co/datasets/liarrrrrrrr/oasst2.oasst2_dpo_pairs_enth
OASST2 DPO Pairs English and Thai
This dataset contains message ChatML. It was create from Open Assistant Conversations Dataset Release 2 (OASST2). You can use to do human preference optimization (DPO, ORPO, and other).
Select Thai only
from datasets import load_dataset
dataset = load_dataset("pythainlp/oasst2_dpo_pairs_enth",split="train")
thai_dataset = dataset.filter(lambda example: example['lang']=="th") # if you want to use English only, change to "en".
license:… See the full description on the dataset page: https://huggingface.co/datasets/pythainlp/oasst2_dpo_pairs_enth.oasst2_orpo_mix_function_call_phi_3_v1
https://huggingface.co/datasets/NickyNicky/function_call_orpo_sft_phi3_chatML_only
https://huggingface.co/datasets/NickyNicky/oasst2_orpo_mix_tokenizer_phi_3_v1
oasst2_french_dpo_pairs
Dataset Card for oasst2_french_dpo_pairs
This dataset was created from OpenAssistant/oasst2 by keeping only the french data and producing dpo pairs with their rank.
Dataset Card Contact
ntnq
oasst2_en_8b_k2_beamsearchoasst2_top4k_en
Dataset Card for "oasst2_top4k_en"
Top ranked conversations of https://huggingface.co/datasets/OpenAssistant/oasst2
Deduplicated, excluded all entried where any two long (>100 chars) answers are similar (cosine sim.>0.8)
Splits: train (4000 entries), test (400 entries)
language-filtered: en
generated using https://github.com/blancsw/deep_4_all/blob/main/datasets/oasst/convert.py
sequence length distribution (X=num. of tokens, Y=frequency)