datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
databricks-dolly-15k
databricks-dolly-15k
This dataset was not originally created by AI Squared. This dataset was curated and created by Databricks.
The below text comes from the original release of the dataset's README file in GitHub (available at https://github.com/databrickslabs/dolly/tree/master/data):
Summary
databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in… See the full description on the dataset page: https://huggingface.co/datasets/aisquared/databricks-dolly-15k.databricks-dolly-15k-cleaned
Summary
databricks-dolly-15k-cleaned is a cleaned up version of the popular databricks-dolly-15k dataset after automatically removing low quality datapoints detected using Cleanlab.
databricks-dolly-15k-context-3k-ragpretrain-databricks-dolly-15kConversion of databricks/databricks-dolly-15k dataset to be used in pretraining.
Python code used for conversion:
from datasets import load_dataset
import pandas
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")
def format(columns):
instruction = columns["instruction"].strip()
answer = columns["response"].strip()
return f"{instruction}\n\n{answer}"pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)
databricks-dolly-15k-cleanset
Summary
databricks-dolly-15k-cleanset can be used to produced CLEANed up versions of the popular databricks-dolly-15k dataSET, which was used to fine-tune the Dolly 2.0. The original databricks-dolly-15k contains 15,000 human-annotated instruction-response pairs covering various categories. However, there are many low-quality responses, incomplete/vague prompts, and other problematic text lurking in the dataset (as with for all real-world instruction tuning datasets). We ran… See the full description on the dataset page: https://huggingface.co/datasets/Cleanlab/databricks-dolly-15k-cleanset.databricks-dolly-15k-jaこのデータセットはkunishou/databricks-dolly-15k-jaを元に作成されています。また、databricks-dolly-15kの情報も参考にしました。
主な修正点
databricks-dolly-15kに注意事項として、注釈は削除した方が良いとの以下記載があり、注釈を削除しています。Reference text (indicated by the context field in the actual dataset) may contain bracketed Wikipedia citation numbers (e.g. [42]) which we recommend users remove for downstream applications.なお注釈の削除については、正規表現を用いた修正を行っております。https://github.com/yuichiro2023/normalize_text
重複した内容の行が複数あり、削除しました。'instruction','input'… See the full description on the dataset page: https://huggingface.co/datasets/morizon/databricks-dolly-15k-ja.databricks-dolly-qa-subset-7kSubset of databricks-dolly-15.
Limited to the following categories: 'open_qa','closed_qa','general_qa'