datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
WebWalkerQA📑 The paper of WebWalkerQA is available at arXiv.
📊 The dataset resource is a collection of 680 questions and answers from the WebWebWalker dataset.
🙋 The dataset is in the form of a JSON file.
The keys in the JSON include:
Question, Answer, Root_Url, and Info. The Info field contains
more detailed information, including Hop, Domain, Language,
Difficulty_Level, Source Website, and Golden_Path.
{
"Question": "When is the paper submission deadline for the ACL 2025 Industry Track, and what… See the full description on the dataset page: https://huggingface.co/datasets/callanwu/WebWalkerQA.webfaqWebFAQ Q&A Dataset
Overview |
Details |
Structure |
Examples |
Considerations |
License |
Citation |
Contact |
Acknowledgement
Overview
The WebFAQ Q&A Dataset is a broad-coverage corpus of 96 million natural question-answer (QA) pairs in 75 languages, gathered from FAQ pages on the web. It leverages structured schema.org FAQPage annotations, making it a unique resource for large-scale Question Answering… See the full description on the dataset page: https://huggingface.co/datasets/PaDaS-Lab/webfaq.webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.OB-News-Websearch
OB News Websearch
100 atomic company-news questions that evaluate web search providers on
a factual lookup task. The model, extract prompt, and runner stay fixed; the
search provider is the variable under test.
This is the public eval set. 100 question/answer pairs. Rows carry the
gold answer, so they are not contamination-free. Use them to inspect the
format and to run a local harness.
Leaderboard
These scores are from the private held-out 300… See the full description on the dataset page: https://huggingface.co/datasets/openbenchmarks/OB-News-Websearch.Vietnamese-nampdn-ai-tiny-webtext-gg-translatedWebWalkerQA📑 The paper of WebWalkerQA is available at arXiv.
📊 The dataset resource is a collection of 680 questions and answers from the WebWebWalker dataset.
🙋 The dataset is in the form of a JSON file.
The keys in the JSON include:
Question, Answer, Root_Url, and Info. The Info field contains
more detailed information, including Hop, Domain, Language,
Difficulty_Level, Source Website, and Golden_Path.
{
"Question": "When is the paper submission deadline for the ACL 2025 Industry Track, and what… See the full description on the dataset page: https://huggingface.co/datasets/shitianqi/WebWalkerQA.web-research-trajectories
Web-Research Agent Trajectories
The first open dataset from Assayo — an open rubric and method for judging the quality
of AI agent trajectories. (The name is from assay*: to test the purity of a metal.)*
An open rubric and a small, hand-built gold set for judging multi-step web-research
agent trajectories. A trajectory is the full record of an agent solving one task by
searching the web, reading sources, and answering with citations — the
think → act → observe → repeat →… See the full description on the dataset page: https://huggingface.co/datasets/Assayo/web-research-trajectories.webcpm_oeTree-of-Web-KnowledgeInspired by Tree of Knowledge (ToK), now remade as Proof of Concept: Tree-of-Web-Knowledge aka ToWK.
Alpaca Dataset created using llama2, Code, Cleaned using score of llm-blender/PairRM and dedup.
Possible improvement: - custom Web search instead of JSON obj by VinciGit00/Scrapegraph-ai.
🔍
.hf-sanitized.hf-sanitized-UDgbtn3GgVkKb3cKXMTHL .img-lbl { position: relative; display: inline-block; cursor: pointer; }
.hf-sanitized.hf-sanitized-UDgbtn3GgVkKb3cKXMTHL .pv { width: 500px; height: auto;… See the full description on the dataset page: https://huggingface.co/datasets/Nekochu/Tree-of-Web-Knowledge.OB-Company-Websearch
OB Company Websearch
10 multi-constraint company-discovery questions with a frozen,
hand-labelled reference set. The public set accompanies the
OpenBenchmarks Multi Turn Company Search Benchmark.
The benchmark holds the research agent and budgets fixed while varying the web
search provider. This release is for the search-only condition: the agent can
search and use result snippets but is not given a page-fetch tool.
Dataset contents
Each question asks for the… See the full description on the dataset page: https://huggingface.co/datasets/openbenchmarks/OB-Company-Websearch.webglm_oeWebWalkerQA📑 The paper of WebWalkerQA is available at arXiv.
📊 The dataset resource is a collection of 680 questions and answers from the WebWebWalker dataset.
🙋 The dataset is in the form of a JSON file.
The keys in the JSON include:
Question, Answer, Root_Url, and Info. The Info field contains
more detailed information, including Hop, Domain, Language,
Difficulty_Level, Source Website, and Golden_Path.
{
"Question": "When is the paper submission deadline for the ACL 2025 Industry Track, and what… See the full description on the dataset page: https://huggingface.co/datasets/shenxiaofan/WebWalkerQA.webGPT_x_dollyThis dataset contains a selection of Q&A-related tasks gathered and cleaned from the webGPT_comparisons set and the databricks-dolly-15k set.
Unicode escapes were explicitly removed, and wikipedia citations in the "output" were stripped through regex to hopefully help any
end-product model ignore these artifacts within their input context.
This data is formatted for use in the alpaca instruction format, however the instruction, input, and output columns are kept separate in
the raw data to… See the full description on the dataset page: https://huggingface.co/datasets/starfishmedical/webGPT_x_dolly.Web-CogBenchThis benchmark was the Web-CogBench mentioned in the paper Web-CogReasoner: Towards Knowledge-Induced Cognitive Reasoning for Web Agents..
The Web-CogBench is used to evaluate Web-CogReasoner, which achieves 84.4 @ Web-CogBench, 86.3 @ VisualWebBench, 30.2% @ WebVoyager, 17.0% and 10.1% @ Online Multimodal-Mind2Web Cross-Tasks and Cross-Webs.
Statistics of the Web-CogBench
Cognition
Task Types
Total Samples
Memorizing
Element… See the full description on the dataset page: https://huggingface.co/datasets/Gnonymous/Web-CogBench.Parameter-Golf-V6-Privacy-Web-Filtering
Solutions Training V6 — Privacy Filtering, Unauthorized Access Triage, and Fast Web Signal Extraction
Overview
V6 extends the V5 auxiliary-training idea into a new direction:
the model should learn to jump over noise and sensitive junk instead of reading or repeating everything.
The dataset trains a signal-first behavior for pages, emails, logs, and incident notes:
skip ads, cookie banners, footers, newsletters, and unrelated chrome,
ignore personal-data-heavy… See the full description on the dataset page: https://huggingface.co/datasets/8Planetterraforming/Parameter-Golf-V6-Privacy-Web-Filtering.web-contentdiscrete_prompting_webqsp
WebQSP Verbalized
This dataset is derived from the WebQSP benchmark and extended with multiple graph-to-text verbalization strategies.It is designed to evaluate how different natural language representations of knowledge graphs affect large language models in knowledge-augmented QA tasks.
Dataset Structure
Splits: train, validation, test
Format: JSONL (one JSON object per line)
web-search-router-labels
Web-Search Need Labels (draft, validation-scale)
Status: draft. This is the audited validation-scale release (517 questions) of a
dataset that labels open-domain questions by whether a mid-size LLM needs web
search to answer them correctly. A train-split scale-up (~509k questions) is in
progress and will be released separately.
Built as part of a semester project on a web-search MCP service and shared
Elasticsearch indexing infrastructure for the Swiss AI initiative… See the full description on the dataset page: https://huggingface.co/datasets/helkhour/web-search-router-labels.Onlineresearch
Computational STEM QA Dataset
Dataset Summary
This dataset contains computationally intensive, self-contained, and unambiguous STEM reasoning problems across Physics, Mathematics, Biology, and Chemistry.
Problems require multi-step reasoning, symbolic manipulation, numerical accuracy, or simulation-based verification. These tasks expose failure modes in state-of-the-art LLMs, making this dataset a strong benchmark for evaluating deep reasoning.
Each example includes:… See the full description on the dataset page: https://huggingface.co/datasets/Web3Survivor/Onlineresearch.MedMCQA
MedMCQA-CoT: 医学多肢選択問題with Chain-of-Thought推論
データセット概要
MedMCQA-CoTは、MedMCQAデータセットの拡張版で、各医学多肢選択問題に高品質なChain-of-Thought（CoT）推論を追加したデータセットです。医学的な推論プロセスを説明できるAIシステムの開発を支援することを目的としています。
主な特徴
2,020件の医学MCQ問題 - 元のMedMCQAデータセットから抽出
Chain-of-Thought推論 - DeepSeek-R1モデルで生成
95.5%の回答精度 - 生成されたCoTが正解に導く割合
0.952の平均品質スコア - 医学用語密度と推論品質に基づく評価
包括的なメタデータ - 品質スコア、医学専門分野、生成統計を含む
データセット詳細
各レコードの構成:
question: MedMCQAからの元の医学問題
answer: 正解の選択肢（A, B, C, D）
cot:… See the full description on the dataset page: https://huggingface.co/datasets/weblab-llm-competition-2025-bridge/MedMCQA.aya-ja-evol-instruct-calm3-dpo-masked
aya-ja-evol-instruct-calm3-dpo-masked
LLMの推論能力の向上のためのデータセット
CohereForAI/aya_datasetの日本語パートを抜粋
Evol-instructを適用し、質問を複雑化
進化元の質問文はこのデータセットに含まれていません。team-hatakeyama-phase2/aya-ja-nemotron-dpo-maskedをご参照ください。
Evol-instructの実装はauthor's repositoryに倣って実装しました。
4種のdepth evolvingと1種のbreadth evolving
chosenカラムはcyberagent/calm3-22b-chatで再生成
rejectedカラムは開発時の内部モデルで生成
29,224件(31,295件の内2,071件削除)ルールベース・機械学習ベースのフィルタリング処理の後、目視確認を行い、個人情報が含まれ得るサンプルについては全件削除対応を実施
Format
{… See the full description on the dataset page: https://huggingface.co/datasets/weblab-GENIAC/aya-ja-evol-instruct-calm3-dpo-masked.pedagogyaya-ja-nemotron-dpo-masked
aya-ja-nemotron-dpo-masked
LLMの推論能力を向上させるためのデータセット
CohereForAI/aya_datasetから日本語パートを抜粋
deepinfraのnvidia/Nemotron-4-340B-Instructで応答を再生成
2024年8月現在ではnvidia/Nemotron-4-340B-Instructは使用不可
5,651件(6,259件の内608件削除)
ルールベース・機械学習ベースのフィルタリング処理の後、目視確認を行い、個人情報が含まれ得るサンプルについては全件削除対応を実施
Format
{
"idx": インデックス,
"prompt": 日本語の指示文,
"chosen": chosenの応答文,
"rejected": rejectedの応答文,
"chosen_model": chosenとしたモデル,
"rejected_model":… See the full description on the dataset page: https://huggingface.co/datasets/weblab-GENIAC/aya-ja-nemotron-dpo-masked.