datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
webglm_datasetpretrain-webglm-qaConversion of THUDM/webglm-qa dataset to be used in pretraining.
Python code used for conversion:
from datasets import load_dataset
import pandas
import re
dataset = load_dataset("THUDM/webglm-qa", split="train")
def format(columns):
return re.sub(r'\[\d\]', '', columns["answer"].strip())
pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)