Felladrin/pretrain-databricks-dolly-15k

Conversion of databricks/databricks-dolly-15k dataset to be used in pretraining. Python code used for conversion: from datasets import load_dataset import pandas dataset = load_dataset("databricks/databricks-dolly-15k", split="train") def format(columns): instruction = columns["instruction"].strip() answer = columns["response"].strip() return f"{instruction}\n\n{answer}" pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)