webglm
Datasets
All datasets matching “webglm”webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.webglm_oeTHUDM__webglm-qa-test-raftChatML-WebGLM-QATHUDM/webglm-qa in ChatML format.
Python code used for conversion:
from datasets import load_dataset
import pandas
import re
import random
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path="Felladrin/Llama-160M-Chat-v1"
)
dataset = load_dataset("THUDM/webglm-qa", split="train")
def format(columns):
references = "\n".join(
[
f"- {columns['references'][i].strip()}"
for i in… See the full description on the dataset page: https://huggingface.co/datasets/Felladrin/ChatML-WebGLM-QA.webglm_datasetwebglm-qa
Dataset Card for "webglm-qa"
More Information needed