datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
officeqa
OfficeQA
Dataset Summary
OfficeQA is a grounded reasoning benchmark by Databricks for evaluating model and agent performance on end-to-end reasoning over real-world documents.
The benchmark consists of question–answer pairs that require reasoning over historical U.S. Treasury Bulletin documents (1939–2025), which contain dense financial tables, charts, and narrative text. OfficeQA is designed to test retrieval, tool use, and multi-step reasoning in… See the full description on the dataset page: https://huggingface.co/datasets/databricks/officeqa.officeqa-pro-v2
OfficeQA Pro v2
Dataset Summary
OfficeQA Pro v2 is a grounded reasoning benchmark by Databricks for evaluating model and agent performance on end-to-end reasoning over real-world documents.
The benchmark consists of question–answer pairs that require reasoning over two centuries of U.S. Federal Accounts of Receipts and Expenditures reporting (1793–2024) — Combined Statements of Receipts, Outlays, and Balances of the United States Government, together with earlier… See the full description on the dataset page: https://huggingface.co/datasets/databricks/officeqa-pro-v2.databricks-dolly-15k
databricks-dolly-15k
This dataset was not originally created by AI Squared. This dataset was curated and created by Databricks.
The below text comes from the original release of the dataset's README file in GitHub (available at https://github.com/databrickslabs/dolly/tree/master/data):
Summary
databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in… See the full description on the dataset page: https://huggingface.co/datasets/aisquared/databricks-dolly-15k.databricks-dolly-15k-cleaned
Summary
databricks-dolly-15k-cleaned is a cleaned up version of the popular databricks-dolly-15k dataset after automatically removing low quality datapoints detected using Cleanlab.
databricks-dolly-15k-context-3k-ragdatabricks-dolly-15k-cleanset
Summary
databricks-dolly-15k-cleanset can be used to produced CLEANed up versions of the popular databricks-dolly-15k dataSET, which was used to fine-tune the Dolly 2.0. The original databricks-dolly-15k contains 15,000 human-annotated instruction-response pairs covering various categories. However, there are many low-quality responses, incomplete/vague prompts, and other problematic text lurking in the dataset (as with for all real-world instruction tuning datasets). We ran… See the full description on the dataset page: https://huggingface.co/datasets/Cleanlab/databricks-dolly-15k-cleanset.databricks-thinking
databricks-thinking
Created by extracing the questions from the [databricks-dolly] dataset and using Qwen3-14b to synthetically generate reasoning traces and answers.
The whole process took 1 day 23 hours 18 minuntes and 8 seconds
Why we created this dataset
The vast majority of publicly available datasets comes from large models such as DeepSeek R1.
The issue with using these large models are obvious: the reasoning traces are extremely long, often longer than the actual… See the full description on the dataset page: https://huggingface.co/datasets/chimbiwide/databricks-thinking.databricksdatabricks-dolly-15k-jaこのデータセットはkunishou/databricks-dolly-15k-jaを元に作成されています。また、databricks-dolly-15kの情報も参考にしました。
主な修正点
databricks-dolly-15kに注意事項として、注釈は削除した方が良いとの以下記載があり、注釈を削除しています。Reference text (indicated by the context field in the actual dataset) may contain bracketed Wikipedia citation numbers (e.g. [42]) which we recommend users remove for downstream applications.なお注釈の削除については、正規表現を用いた修正を行っております。https://github.com/yuichiro2023/normalize_text
重複した内容の行が複数あり、削除しました。'instruction','input'… See the full description on the dataset page: https://huggingface.co/datasets/morizon/databricks-dolly-15k-ja.pretrain-databricks-dolly-15kConversion of databricks/databricks-dolly-15k dataset to be used in pretraining.
Python code used for conversion:
from datasets import load_dataset
import pandas
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")
def format(columns):
instruction = columns["instruction"].strip()
answer = columns["response"].strip()
return f"{instruction}\n\n{answer}"pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)
databricks-dolly-qa-subset-7kSubset of databricks-dolly-15.
Limited to the following categories: 'open_qa','closed_qa','general_qa'