datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.webglm_oeTHUDM__webglm-qa-test-raftChatML-WebGLM-QATHUDM/webglm-qa in ChatML format.
Python code used for conversion:
from datasets import load_dataset
import pandas
import re
import random
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path="Felladrin/Llama-160M-Chat-v1"
)
dataset = load_dataset("THUDM/webglm-qa", split="train")
def format(columns):
references = "\n".join(
[
f"- {columns['references'][i].strip()}"
for i in… See the full description on the dataset page: https://huggingface.co/datasets/Felladrin/ChatML-WebGLM-QA.webglm_datasetwebglm-qa
Dataset Card for "webglm-qa"
More Information needed
webglm_vi
Dataset Card for "webglm_vi"
More Information needed
THUDM__webglm-qa-train-raft-3pretrain-webglm-qaConversion of THUDM/webglm-qa dataset to be used in pretraining.
Python code used for conversion:
from datasets import load_dataset
import pandas
import re
dataset = load_dataset("THUDM/webglm-qa", split="train")
def format(columns):
return re.sub(r'\[\d\]', '', columns["answer"].strip())
pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)
THUDM__webglm-qa-train-raft-2THUDM__webglm-qa-train-raftwebglm-qawebglm-harshwebglm-data-100