datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
databricks-dolly-15k
Summary
databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several
of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification,
closed QA, generation, information extraction, open QA, and summarization.
This dataset can be used for any purpose, whether academic or commercial, under the terms of the
Creative Commons Attribution-ShareAlike 3.0 Unported… See the full description on the dataset page: https://huggingface.co/datasets/databricks/databricks-dolly-15k.chinese-dolly-15kChinese-Dolly-15k是骆驼团队翻译的Dolly instruction数据集
最后49条数据因为翻译长度超过限制，没有翻译成功，建议删除或者手动翻译一下
原来的数据集'databricks/databricks-dolly-15k'是由数千名Databricks员工根据InstructGPT论文中概述的几种行为类别生成的遵循指示记录的开源数据集。这几个行为类别包括头脑风暴、分类、封闭型问答、生成、信息提取、开放型问答和摘要。
在知识共享署名-相同方式共享3.0（CC BY-SA 3.0）许可下，此数据集可用于任何学术或商业用途。
我们会陆续将更多数据集发布到hf，包括
Coco Caption的中文翻译
CoQA的中文翻译
CNewSum的Embedding数据
增广的开放QA数据
WizardLM的中文翻译
MMC4的中文翻译
如果你也在做这些数据集的筹备，欢迎来联系我们，避免重复花钱。
骆驼(Luotuo): 开源中文大语言模型
https://github.com/LC1332/Luotuo-Chinese-LLM… See the full description on the dataset page: https://huggingface.co/datasets/silk-road/chinese-dolly-15k.databricks-dolly-15k-ja
databricks-dolly-15k-ja
This repository provides an instruction tuning dataset developed by LLM-jp, a collaborative project launched in Japan.
This dataset is a Japanese translation of databricks-dolly-15k using DeepL.
Send Questions to
llm-jp(at)nii.ac.jp
Model Card Authors
The names are listed in alphabetical order.
Hirokazu Kiyomaru, Hiroshi Matsuda, Jun Suzuki, Namgi Han, Saku Sugawara, Shota Sasaki, Shuhei Kurita, Taishi Nakamura, Takashi Kodama, Takumi… See the full description on the dataset page: https://huggingface.co/datasets/llm-jp/databricks-dolly-15k-ja.databricks-dolly-15k-trThis dataset is machine-translated version of databricks-dolly-15k.jsonl into Turkish.
Used googletrans==3.1.0a0 to translation.
dolly-15k-libretranslate-pt
Summary
databricks-dolly-15k ( https://huggingface.co/datasets/databricks/databricks-dolly-15k/ ) is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
This is a portuguese translation done with libretranslate (… See the full description on the dataset page: https://huggingface.co/datasets/Gustrd/dolly-15k-libretranslate-pt.dolly-15k-dutch
Dataset Card for Dolly 15k Dutch
Dataset Summary
This dataset contains 14,934 instructions, contexts and responses, in several natural language categories such as classification, closed QA, generation, etc. The English original dataset was created by @databricks, who crowd-sourced the data creation via its employees. The current dataset is a translation of that dataset through ChatGPT (gpt-3.5-turbo).
☕ Want to help me out? Translating the data with the OpenAI API, and… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/dolly-15k-dutch.ChatML-webGPT_x_dollystarfishmedical/webGPT_x_dolly in ChatML format, ready to use in HuggingFace TRL's SFT Trainer.
Python code used for conversion:
from datasets import load_dataset
from transformers import AutoTokenizer
import random
tokenizer = AutoTokenizer.from_pretrained("Felladrin/Llama-160M-Chat-v1")
dataset = load_dataset("starfishmedical/webGPT_x_dolly", split="train")
def format(columns):
instruction = columns["instruction"].strip()
input = columns["input"].strip()
assistant_message =… See the full description on the dataset page: https://huggingface.co/datasets/Felladrin/ChatML-webGPT_x_dolly.dolly-15k-chinese-zhtw
Dataset Card for "dolly-15k-chinese-zhtw"
內容
dolly-15k-chinese-zhtw 是一個開源數據集，它的原始數據集 databricks-dolly-15k 包含由數千名 Databricks 員工產生的指令追蹤記錄，涉及 InstructGPT 論文中概述的幾個行為類別，包括腦力激盪、分類、封閉式QA、生成、資訊擷取、開放式QA 和總結。
根據以下條款，該資料集可用於任何目的，無論是學術目的還是商業目的 Creative Commons Attribution-ShareAlike 3.0 Unported License。
支援的任務
訓練 LLMs
合成數據的生成
數據增強
概述
databricks-dolly-15k 是由數千名 Databricks 員工產生的超過 15,000 筆記錄的語料庫，使大型語言模型能夠展現 ChatGPT 的神奇互動性。 Databricks… See the full description on the dataset page: https://huggingface.co/datasets/erhwenkuo/dolly-15k-chinese-zhtw.dolly-15k-pirate-speechDataset for writing style transfer experimentation based on article:
https://ai-r.com/blog/pirate-linguistics-and-tone-of-voice-fine-tuning-llms-to-talk-like-swashbucklers
Only responses are in 'pirate speech'
arrr python library was used to simply change original responses to 'pirate speech' responses
https://pypi.org/project/arrr/
dolly-15k-rlhf-instructgpt-format
Dolly 15k RLHF Datasets in InstructGPT Format
This repository packages databricks/databricks-dolly-15k into three RLHF-oriented
dataset configurations inspired by the InstructGPT data flow:
sft: supervised fine-tuning examples with prompt, completion, and text.
rm_schema: reward-modeling schema/prompt pool with empty chosen and rejected
fields, reference_response, and ready_for_rm=false.
rm_synthetic: reward-modeling proxy pairs where Dolly reference_response is
used as chosen and… See the full description on the dataset page: https://huggingface.co/datasets/costadev00/dolly-15k-rlhf-instructgpt-format.webGPT_x_dollyThis dataset contains a selection of Q&A-related tasks gathered and cleaned from the webGPT_comparisons set and the databricks-dolly-15k set.
Unicode escapes were explicitly removed, and wikipedia citations in the "output" were stripped through regex to hopefully help any
end-product model ignore these artifacts within their input context.
This data is formatted for use in the alpaca instruction format, however the instruction, input, and output columns are kept separate in
the raw data to… See the full description on the dataset page: https://huggingface.co/datasets/starfishmedical/webGPT_x_dolly.ChatML-databricks-dolly-15kdatabricks/databricks-dolly-15k in ChatML format.
Python code used for conversion:
from datasets import load_dataset
import pandas
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path="Felladrin/Llama-160M-Chat-v1"
)
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")
def format(columns):
instruction = columns["instruction"].strip()
context = columns["context"].strip()
response =… See the full description on the dataset page: https://huggingface.co/datasets/Felladrin/ChatML-databricks-dolly-15k.databricks-dolly-15k-uk
Summary
databricks-dolly-15k-uk is an open source dataset based on databricks/databricks-dolly-15k instruction-following dataset, but machine translated using facebook/m2m100_1.2B model.Tasks covered include brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.Expect this dataset to not be grammatically correct and having obvious pitfalls of machine translation.
Original Summary
# Summary
`databricks-dolly-15k` is an open… See the full description on the dataset page: https://huggingface.co/datasets/robinhad/databricks-dolly-15k-uk.databricksdatabricks-dolly-15k-chinese
Dataset Summary
🏡🏡🏡🏡Fine-tune Dataset:中文数据集🏡🏡🏡🏡
😀😀😀😀😀😀😀😀 这个数据集是databricks/databricks-dolly-15k的中文版本，是直接翻译过来，没有经过人为检查语法。 对databricks/databricks-dolly-15k的描述，请看他的dataset card。
😀😀😀😀😀😀😀😀 This data set is the Chinese version of databricks/databricks-dolly-15k, which is directly translated without human-checked grammar. For a description of databricks/databricks-dolly-15k, see its dataset card.
dolly_hhrlhf
Dataset Card for "dolly_hhrlhf"
This is the dataset from mosaic mosaicml/dolly_hhrlhf removing some duplicates found.
More Information needed
ko.databricks-dolly-15k원본 데이터셋: databricks/databricks-dolly-15k
databricks-dolly-15k-th
Summary
This is a Thai 🇹🇭-instructed dataset translated from databricks-dolly-15k using Google Cloud Translation.
databricks-dolly-15k is an open-source dataset of instruction-following records generated by thousands of Databricks employees in several behavioral
categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
This dataset can be used for any purpose, whether academic… See the full description on the dataset page: https://huggingface.co/datasets/Thaweewat/databricks-dolly-15k-th.KenithZ-dolly-zh-51k
Dolly中文训练集
基于Chinese-LLaMA-Alpaca的转换成的dolly数据集
需要做的事情
将alpaca_data_zh_51k.json数据集转换为databricks-dolly-15k.jsonl数据集的格式
转换后的数据集集需要手动补充category（正在进行）
修正原作者从chatGPT爬取的语义不通或数据错误的指令数据（正在进行）
databricks-dolly-15k-ja-scoredFor the English version, please click here.
概要
databricks-dolly-15k-ja-scoredはkunishou/databricks-dolly-15k-jaの派生であり、BERTScoreによって提供される翻訳品質スコアが追加されています。
このデータセットは、学術的・商業的問わずクリエイティブ・コモンズ 表示 - 継承 3.0 非移植ライセンスの条件の下で何にでも使用することができます。
翻訳の品質スコア
databricks-dolly-15k-jaは、databricks-dolly-15kを機械翻訳したものです。databricks-dolly-15k-jaに含まれるデータを調べてみると、以下のような品質の悪いデータが存在することが分かりました。
inputとoutputが全く同じであるデータ
outputがinstructionにコピーされているデータ
表記ゆれによって表現の一貫性が保たれていないデータ
固有名詞などの翻訳に失敗しているデータ… See the full description on the dataset page: https://huggingface.co/datasets/sakusakumura/databricks-dolly-15k-ja-scored.databricks-dolly-15k-tamildolly3k_ca
Dataset Card for dolly3k_ca
dolly3k_ca is a question answering dataset in Catalan, professionally translated from a filtered version of databricks-dolly-15k dataset in English.
Dataset Details
Dataset Description
dolly3k_ca (Dolly 3K instances - Catalan) is based on question-answer instance pairs written by humans. The dataset consists of 3232 instances in the train split. Each instance contains an instruction or question, and one answer. Every instance is… See the full description on the dataset page: https://huggingface.co/datasets/projecte-aina/dolly3k_ca.transformed_JSON_databricks-dolly-15k.jsonl
Transformed Databricks-Dolly-15k Dataset
Summary
The Transformed Databricks-Dolly-15k dataset is a modification of the original open-source dataset created by Databricks employees, designed to facilitate instruction-following abilities in large language models (LLMs). This version has been specifically adapted to include responses in a JSON format, enhancing its utility for tasks requiring structured output.
Modifications
The primary transformation applied to… See the full description on the dataset page: https://huggingface.co/datasets/ramachetan22/transformed_JSON_databricks-dolly-15k.jsonl.Alpaca_Dollytest-dolly-15ja-for-stftrainer
Dataset Card for Dataset Name
Dataset Summary
It is just a dataset of dolly-15k-jp(*1) converted to jsonl form so that it can be used in SFTTrainer(*2)'s dataset_text_field property.
(*1)https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja(*2)https://huggingface.co/docs/trl/main/en/sft_trainer
Languages
ja
Licensing Information
This dataset is licensed under CC BY SA 3.0
Special… See the full description on the dataset page: https://huggingface.co/datasets/Coaso/test-dolly-15ja-for-stftrainer.dolly-14k-ines
dolly-14k-ines
Description
The dolly-14k-ines dataset is derived from the databricks-dolly-15k-ja, which is a machine-translated version of the databricks/dolly-15k. The entries have been further converted to reflect the speech pattern of Ines Fujin, a character from "Umamusume Pretty Derby." The conversion process utilized a model specifically developed for this task, and entries that did not successfully emulate the character's textual speech style were omitted… See the full description on the dataset page: https://huggingface.co/datasets/sakusakumura/dolly-14k-ines.dolly-15k-turkmen
Turkmen Dolly 15k Dataset
Overview
This dataset is a Turkmen translation of the original Dolly 15k dataset. The Dolly dataset is a publicly available instruction-following dataset created by Databricks, containing 15,000 high-quality human-generated prompt-response pairs. This Turkmen version aims to extend the accessibility of instruction-following datasets to the Turkmen language community.
Dataset Details
Original Dataset: Dolly 15k
Language: Turkmen
Number… See the full description on the dataset page: https://huggingface.co/datasets/mamed0v/dolly-15k-turkmen.rationale-databricks-dolly-cqa
Dataset Overview
Filtered and annotated version of the closed-question answering part (~1.5k datapoints) of the Databricks Dolly Dataset intended for the task of rationale extraction.
Citation
@article{pirenne2024exploration,
title={Exploration of Closed-Domain Question Answering Explainability Methods With a Sentence-Level Rationale Dataset},
author={Pirenne, Lize and Mokeddem, Samy and Ernst, Damien and Louppe, Gilles},
year={2024}
}… See the full description on the dataset page: https://huggingface.co/datasets/Inversta/rationale-databricks-dolly-cqa.dolly15k_hinglish_dataset_cleaneddatabricks-dolly-15k-azThis dataset is a machine-translated version of databricks-dolly-15k.jsonl into Azerbaijani. Dataset size is 8k.
Summary
databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several
of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification,
closed QA, generation, information extraction, open QA, and summarization.
This dataset can be used for any purpose… See the full description on the dataset page: https://huggingface.co/datasets/w95/databricks-dolly-15k-az.databricks-dolly-8k-qa-open-close