datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
oasst1
OpenAssistant Conversations Dataset (OASST1)
Dataset Summary
In an effort to democratize research on large-scale alignment, we release OpenAssistant
Conversations (OASST1), a human-generated, human-annotated assistant-style conversation
corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292
quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus
is a product of a worldwide crowd-sourcing effort… See the full description on the dataset page: https://huggingface.co/datasets/OpenAssistant/oasst1.oasst1_pairwise_rlhf_reward
Dataset Card for "oasst1_pairwise_rlhf_reward"
OASST1 dataset preprocessed for reward modeling:
import pandas as pd
from datasets import load_dataset,concatenate_datasets, Dataset, DatasetDict
import numpy as np
dataset = load_dataset("OpenAssistant/oasst1")
df=concatenate_datasets(list(dataset.values())).to_pandas()
m2t=df.set_index("message_id")['text'].to_dict()
m2r=df.set_index("message_id")['role'].to_dict()
m2p=df.set_index('message_id')['parent_id'].to_dict()… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward.german-oasst1-qa
License & Attribution
MTEB-format derivative of AgentWaller/german-oasst1-qa-format (German OpenAssistant OASST1, QA format). Query = question; corpus = answer. Licensed under Apache-2.0 (same as source).
oasst1-89k-ja
This dataset was created by automatically translating "OpenAssistant/oasst1" into Japanese.
The "ng_translation" flag indicates that the translation was not successful, and "1" means that the translation failed.Therefore, for data with "1", "text" and "text_en" contain the same text.
Update:
2023/11/12oasst1-89k-jaをチャット形式に変換したoasst1-chat-44k-jaを公開しました。
2023/10/21自動翻訳によるコード関連データの翻訳誤り2000箇所程度を手動で修正しました。
修正イメージを表示
修正前
もちろん！これは、Flask… See the full description on the dataset page: https://huggingface.co/datasets/kunishou/oasst1-89k-ja.oasst1_orignal_de
Dataset Card for "oasst1_orignal_de"
More Information needed
oasst1_th
Dataset Card for "oasst1_th"
More Information needed
oasst1-21k-ja
oasst1-21k-ja
This repository provides an instruction tuning dataset developed by LLM-jp, a collaborative project launched in Japan.
This dataset is a Japanese translation of an English subset of oasst1 using DeepL.
English subset is here.
Send Questions to
llm-jp(at)nii.ac.jp
Model Card Authors
The names are listed in alphabetical order.
Hirokazu Kiyomaru, Hiroshi Matsuda, Jun Suzuki, Namgi Han, Saku Sugawara, Shota Sasaki, Shuhei Kurita, Taishi… See the full description on the dataset page: https://huggingface.co/datasets/llm-jp/oasst1-21k-ja.oasst1_ru_main_branch
Based on oasst1.
Only Russian message trees, only main branches.
Script: get_oasst_ru.py
oasst1_ca
Dataset Card for oasst1_ca
oasst1_ca is a conversational dataset in Catalan that has been professionally translated from the OASST1 dataset.
Dataset Details
Dataset Description
oasst1_ca (OpenAssistant Conversations Release 1 - Catalan) consists of human-generated, human-annotated assistant-style conversation corpus. It includes 5213 messages in the train split and 273 messages in the validation split. To arrive to this number, we filter the dataset (See… See the full description on the dataset page: https://huggingface.co/datasets/projecte-aina/oasst1_ca.oasst1_dense_flat
Dataset Card for "oasst1_dense_flat"
OASST1 dataset
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)
from datasets import Dataset, DatasetDict
d={}
for split in ['train','validation']:
df=load_dataset("OpenAssistant/oasst1")[split].to_pandas()
m2t=df.set_index("message_id")['text'].to_dict()
df['parent_text']=df.parent_id.map(lambdax: m2t.get(x,''))
df=df[df.labels.map(lambda x:x!=None)]… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/oasst1_dense_flat.oasst1-chat-44k-jaoasst1-89k-jaをチャット形式に変換したデータセットになります。マルチターン会話でのファインチューニングをする際にご活用下さい（1レコードのトークン長が大きいのでそれなりの計算リソースが必要になります）。フォーマットは ShareGPT 形式になっています。ファインチューニングをする際はこちらの記事を参考にして下さい。
oasst1-ja-89k Repositoryhttps://github.com/kunishou/oasst1-89k-ja
OpenAssistant/oasst1https://huggingface.co/datasets/OpenAssistant/oasst1
oasst1-oasst2-ja_single_dedupedoasst1
OpenAssistant Conversations Dataset (OASST1)
Dataset Summary
In an effort to democratize research on large-scale alignment, we release OpenAssistant
Conversations (OASST1), a human-generated, human-annotated assistant-style conversation
corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292
quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus
is a product of a worldwide crowd-sourcing effort… See the full description on the dataset page: https://huggingface.co/datasets/Tylener/oasst1.details_h2oai__h2ogpt-gm-oasst1-multilang-1024-20b
Dataset Card for Evaluation run of h2oai/h2ogpt-gm-oasst1-multilang-1024-20b
Dataset Summary
Dataset automatically created during the evaluation run of model h2oai/h2ogpt-gm-oasst1-multilang-1024-20b on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_h2oai__h2ogpt-gm-oasst1-multilang-1024-20b.oasst1-en-hun-gemini
Open assistant 1 dataset hungarian translation (english subset)
This dataset contains hungarian translations for the oasst1 dataset's english subset. The translations were done via gemini pro and the model was instructed to keep stlye, meaning and english entites as they are. I think this produced a higher quality translation than google translate, but even this version is far from perfect.
The exact code used for creating the dataset can be found here.
license:… See the full description on the dataset page: https://huggingface.co/datasets/jazzysnake01/oasst1-en-hun-gemini.oasst1
OpenAssistant Conversations Dataset (OASST1)
Dataset Summary
In an effort to democratize research on large-scale alignment, we release OpenAssistant
Conversations (OASST1), a human-generated, human-annotated assistant-style conversation
corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292
quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus
is a product of a worldwide crowd-sourcing effort… See the full description on the dataset page: https://huggingface.co/datasets/BrunoN-Dev/oasst1.oasst1
OpenAssistant Conversations Dataset (OASST1)
Dataset Summary
In an effort to democratize research on large-scale alignment, we release OpenAssistant
Conversations (OASST1), a human-generated, human-annotated assistant-style conversation
corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292
quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus
is a product of a worldwide crowd-sourcing effort… See the full description on the dataset page: https://huggingface.co/datasets/Theafricatechguy/oasst1.oasst1-decontaminated
Decontaminated — OpenAssistant/oasst1
What this is
A filtered version of OpenAssistant/oasst1 (revision
fdf72ae0827c1cda404aff25b6603abec9e3399b) with exact-duplicate rows and rows overlapping standard benchmark test sets
removed. This is a different artifact from the companion contamination report — that one is an
audit of what's wrong; this one is the corpus with those rows actually taken out, ready to train on.
Processing
Deduplicated… See the full description on the dataset page: https://huggingface.co/datasets/liodon-ai/oasst1-decontaminated.openassistant_oasst1_h2ogpt
h2oGPT Data Card
Summary
H2O.ai's openassistant_oasst1_h2ogpt is an open-source instruct-type dataset for fine-tuning of large language models, licensed for commercial use.
Number of rows: 48307
Number of columns: 3
Column names: ['input', 'prompt_type', 'source']
Source
Original Open Assistant data in tree structure
This flattened dataset created by script in h2oGPT repository
oasst1
Dataset Card for "oasst1"
More Information needed
oasst1_euLlama-2-databricks-dolly-oasst1-es-lower-1024-tokens
Llama-2-databricks-dolly-oasst1-es-lower-1024-tokens
Union of https://huggingface.co/datasets/dariolopez/Llama-2-databricks-dolly-es and https://huggingface.co/datasets/dariolopez/Llama-2-oasst1-es
Filtering of texts with less than 1024 tokens.
german-oasst1-qa-formatArabic_guanaco_oasst1
Dataset Card for "Arabic_guanaco_oasst1"
This dataset is the openassistant-guanaco dataset a subset of the Open Assistant dataset translated to Arabic.
You can find the original dataset here: https://huggingface.co/datasets/timdettmers/openassistant-guanaco
Or the main dataset here: https://huggingface.co/datasets/OpenAssistant/oasst1/tree/main
This subset of the data only contains the highest-rated paths in the conversation tree, with a total of 9,846 samples.
For further… See the full description on the dataset page: https://huggingface.co/datasets/alielfilali01/Arabic_guanaco_oasst1.oasst1-tiny-subset
Dataset Card for "oasst1-tiny-subset"
More Information needed
details_habanoz__tinyllama-oasst1-top1-instruct-full-lr1-5-v0.1
Dataset Card for Evaluation run of habanoz/tinyllama-oasst1-top1-instruct-full-lr1-5-v0.1
Dataset Summary
Dataset automatically created during the evaluation run of model habanoz/tinyllama-oasst1-top1-instruct-full-lr1-5-v0.1 on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_habanoz__tinyllama-oasst1-top1-instruct-full-lr1-5-v0.1.oasst1-contamination-report
Contamination Report — OpenAssistant/oasst1
What this is
A row-level audit of OpenAssistant/oasst1 (revision
fdf72ae0827c1cda404aff25b6603abec9e3399b) for exact 13-gram overlap with standard benchmark test sets
(gsm8k, hellaswag, humaneval, mmlu). This is not a filtered copy of the source — it's a new
artifact: a list of which rows overlap which benchmark, plus summary statistics, so anyone
training on the source can decide how to handle it.… See the full description on the dataset page: https://huggingface.co/datasets/liodon-ai/oasst1-contamination-report.oasst1-ar-threadsopenassistant_oasst1_h2ogpt_graded
h2oGPT Data Card
Summary
H2O.ai's openassistant_oasst1_h2ogpt_graded is an open-source instruct-type dataset for fine-tuning of large language models, licensed for commercial use.
Number of rows: 30368
Number of columns: 5
Column names: ['input', 'source', 'prompt_type', 'grade_deberta', 'id']
Source
Original Open Assistant data in tree structure
This flattened dataset created by script in h2oGPT repository
oasst1_de