datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
WebCompass
WebCompass
A unified multimodal benchmark for evaluating LLMs' ability to generate, edit, and repair functional web pages. WebCompass spans three input modalities — text design documents, reference screenshots, and video demonstrations — and three task families — generation, editing, and repair.
GitHub: NJU-LINK/WebCompass
Project Page: nju-link.github.io/WebCompass
Quick Start
from datasets import load_dataset
# Generation tasks (existing)
ds_text =… See the full description on the dataset page: https://huggingface.co/datasets/NJU-LINK/WebCompass.webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
WangchanLION-Web
Citation
@misc{phatthiyaphaibun2025mangosteenopenthaicorpus,
title={Mangosteen: An Open Thai Corpus for Language Model Pretraining},
author={Wannaphong Phatthiyaphaibun and Can Udomcharoenchaikit and Pakpoom Singkorapoom and Kunat Pipatanakul and Ekapol Chuangsuwanich and Peerat Limkonchotiwat and Sarana Nutanong},
year={2025},
eprint={2507.14664},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2507.14664},
}
We… See the full description on the dataset page: https://huggingface.co/datasets/aisingapore/WangchanLION-Web.Agent-G2-ALFWorld-Webshop-sft-data
Agent-G2 SFT Data
Agent-G2 SFT Data contains reasoning and action trajectories for supervised
fine-tuning (SFT) in the Agent-G2 project.
Associated paper: Agent-G2: Gaussian Guidance for Agentic Reinforcement
Learning — accepted to the EMNLP 2026 Main Conference.
The dataset covers two interactive agent environments:
WebShop: agents search for products, select options, and complete
purchases according to user requirements.
ALFWorld: agents interact with household environments… See the full description on the dataset page: https://huggingface.co/datasets/xiamoent/Agent-G2-ALFWorld-Webshop-sft-data.WebWorldData
WebWorldData 🌐
Overview
WebWorldData is a large-scale dataset of 1.06M web interaction trajectories collected from the open web, designed for training browser world models. It is the training data behind the WebWorld model series.
Each trajectory consists of sequences of (state, action, next_state) transitions, where states are represented as A11y Trees extracted from real websites using Playwright.
Dataset Statistics
Total… See the full description on the dataset page: https://huggingface.co/datasets/Qwen/WebWorldData.webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
Odia-Web-Corpus-v1
Odia Web Corpus v1
The inaugural release of a curated Odia (Oriya) web text corpus. Scraped and filtered from publicly accessible web sources to support Odia language modeling and NLP research.
Dataset Details
Language: Odia (Oriya, ISO 639-3: ory)
Format: JSONL (one JSON object per line)
Size: ~650K documents, ~0.9 GB text
License: CC-BY-4.0
Data Fields
Field
Type
Description
text
string
Cleaned document body
title
string… See the full description on the dataset page: https://huggingface.co/datasets/saidutta69/Odia-Web-Corpus-v1.WebNovels-Ja
WebNovels-Ja
日本語はこちら
このデータセットは、著作権法第三十条の四の情報解析（機械学習等）の目的でのみ使用が許可されています。それ以外の用途での使用はライセンスにより禁止されています。
This dataset is only permitted for use under Article 30-4 of the Copyright Law of Japan for data analysis (such as machine learning) purposes. Any use for purposes other than those specified is prohibited by the license.
Dataset Summary
This is a dataset of Japanese web novels that I personally collected.
Total number of characters: approximately 10 billion… See the full description on the dataset page: https://huggingface.co/datasets/OmniAICreator/WebNovels-Ja.Vietnamese-nampdn-ai-tiny-webtext-gg-translatedOB-Code-Websearch
Web Search for Coding Agents
30 realistic product tickets that evaluate web search providers in a coding-agent workflow. The model, task, budgets, and runner stay fixed; the search or fetch provider is the variable under test.
This is the public dataset. It is 30 tickets, 10 per vendor, one
source page each. The
boards are scored on a held-out private set that is not distributed, so
that vendors and models cannot train and fit to the benchmark. Use these rows
to… See the full description on the dataset page: https://huggingface.co/datasets/openbenchmarks/OB-Code-Websearch.basilisk-webpentest
Basilisk WebPentest Dataset
Instruction‑tuning data for web penetration testing, used to train sl4de/Basilisk-7B.
Direct, uncensored expert answers (payloads, PoCs, tool commands, reports) for authorized security testing and education.
Files
File
Records
Domains
basilisk_v0.jsonl
2,999
SQLi, XSS, Recon
basilisk_v1.jsonl
8,054
19 (v0 + SSRF, XXE, deserialization, auth/JWT, access control, API, SSTI, LFI, CSRF, CORS, request smuggling, prototype… See the full description on the dataset page: https://huggingface.co/datasets/sl4de/basilisk-webpentest.web-research-trajectories
Web-Research Agent Trajectories
The first open dataset from Assayo — an open rubric and method for judging the quality
of AI agent trajectories. (The name is from assay*: to test the purity of a metal.)*
An open rubric and a small, hand-built gold set for judging multi-step web-research
agent trajectories. A trajectory is the full record of an agent solving one task by
searching the web, reading sources, and answering with citations — the
think → act → observe → repeat →… See the full description on the dataset page: https://huggingface.co/datasets/Assayo/web-research-trajectories.WEBPRMBENCH
WebPRMBench
The first comprehensive evaluation benchmark for Web Process Reward Models
Published at ICLR 2026
Paper | Code | Website | Collection | Demo
Overview
WebPRMBench is the first comprehensive evaluation benchmark dedicated to Web Process Reward Models (WebPRMs). It evaluates how well a reward model can judge the quality of web agent actions during long-horizon web navigation. Each instance presents a web state (page context, trajectory history, user… See the full description on the dataset page: https://huggingface.co/datasets/ZYao720/WEBPRMBENCH.MM-WebGen-Bench
MM-WebGen-Bench: A Benchmark for Multimodal Webpage Generation
MM-WebGen-Bench is a multi-level evaluation benchmark for multimodal webpage generation, proposed in MM-WebAgent. It contains 120 curated webpage design prompts covering 11 scene categories, 11 visual styles, and diverse multimodal compositions (4 video types, 8 image types, and 17 chart types).
Links
Project Page: aka.ms/mm-webagent
GitHub: microsoft/MM-webagent
Paper: MM-WebAgent: A Hierarchical… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/MM-WebGen-Bench.WebCPM_WK
Dataset Card for WebCPM_WK
Dataset Summary
本数据集是由我们对WebCPM的pipeline数据进行二次处理之后构建而成。
主要包括过滤原始数据中的一些低质量数据，使用GPT4和ChatGPT扩充原始数据，以及使用随机替换、拼接的方式增强原始数据。
该数据集主要的目的是通过指令微调的方式提高LLM的两个能力：
给定问题和文档，抽取文档中与问题相关知识的能力。
给定参考材料和问题，根据参考材料回答问题的能力。
Licensing Information
The dataset is available under the Creative Commons NonCommercial (CC BY-NC 4.0).
lemonseed-uni-web
lemonseed-uni-web
LemonSeed — professional web prose (fineweb-edu + refinedweb).
Format
JSON Lines (.jsonl), one example per line.
Provenance & License
Derived from HuggingFaceFW/fineweb-edu sample-10BT (ODC-By) + mattymchen/refinedweb-3m (no stated license).
webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
lumos_web_agent_plan_iterative
🪄 Agent Lumos: Unified and Modular Training for Open-Source Language Agents
🌐[Website]
📝[Paper]
🤗[Data]
🤗[Model]
🤗[Demo]
We introduce 🪄Lumos, Language Agents with Unified Formats, Modular Design, and Open-Source LLMs. Lumos unifies a suite of complex interactive tasks and achieves competitive performance with GPT-4/3.5-based and larger open-source agents.
Lumos has following features:
🧩 Modular Architecture:
🧩 Lumos consists of planning, grounding… See the full description on the dataset page: https://huggingface.co/datasets/ai2lumos/lumos_web_agent_plan_iterative.Tree-of-Web-KnowledgeInspired by Tree of Knowledge (ToK), now remade as Proof of Concept: Tree-of-Web-Knowledge aka ToWK.
Alpaca Dataset created using llama2, Code, Cleaned using score of llm-blender/PairRM and dedup.
Possible improvement: - custom Web search instead of JSON obj by VinciGit00/Scrapegraph-ai.
🔍
.hf-sanitized.hf-sanitized-UDgbtn3GgVkKb3cKXMTHL .img-lbl { position: relative; display: inline-block; cursor: pointer; }
.hf-sanitized.hf-sanitized-UDgbtn3GgVkKb3cKXMTHL .pv { width: 500px; height: auto;… See the full description on the dataset page: https://huggingface.co/datasets/Nekochu/Tree-of-Web-Knowledge.webXOS_chess_RL
_______ ______ _______ _______ _______ _______ _______ _______ _______ _
|\ /|( ____ \( ___ \ |\ /|( ___ )( ____ \ ( ____ \|\ /|( ____ \( ____ \( ____ \ ( ____ )( \
| ) ( || ( \/| ( ) )( \ / )| ( ) || ( \/ | ( \/| ) ( || ( \/| ( \/| ( \/ | ( )|| (
| | _ | || (__ | (__/ / \ (_) / | | | || (_____ | | | (___) || (__ | (_____ | (_____ | (____)|| |
|… See the full description on the dataset page: https://huggingface.co/datasets/webxos/webXOS_chess_RL.lumos_web_agent_ground_iterative
🪄 Agent Lumos: Unified and Modular Training for Open-Source Language Agents
🌐[Website]
📝[Paper]
🤗[Data]
🤗[Model]
🤗[Demo]
We introduce 🪄Lumos, Language Agents with Unified Formats, Modular Design, and Open-Source LLMs. Lumos unifies a suite of complex interactive tasks and achieves competitive performance with GPT-4/3.5-based and larger open-source agents.
Lumos has following features:
🧩 Modular Architecture:
🧩 Lumos consists of planning, grounding… See the full description on the dataset page: https://huggingface.co/datasets/ai2lumos/lumos_web_agent_ground_iterative.Parameter-Golf-V6-Privacy-Web-Filtering
Solutions Training V6 — Privacy Filtering, Unauthorized Access Triage, and Fast Web Signal Extraction
Overview
V6 extends the V5 auxiliary-training idea into a new direction:
the model should learn to jump over noise and sensitive junk instead of reading or repeating everything.
The dataset trains a signal-first behavior for pages, emails, logs, and incident notes:
skip ads, cookie banners, footers, newsletters, and unrelated chrome,
ignore personal-data-heavy… See the full description on the dataset page: https://huggingface.co/datasets/8Planetterraforming/Parameter-Golf-V6-Privacy-Web-Filtering.webui
🌐 WebUI
A high-quality dataset for training AI models to understand and generate modern websites.
The dataset contains structured webpage samples including HTML, screenshots, UI element annotations, semantic labels, color palettes, fonts, layout metadata, and accessibility information. Each sample represents a complete webpage that can be used for web generation, UI understanding, or multimodal training. :contentReference[oaicite:0]{index=0}
✨ Features
📄… See the full description on the dataset page: https://huggingface.co/datasets/xlelords/webui.discrete_prompting_webqsp
WebQSP Verbalized
This dataset is derived from the WebQSP benchmark and extended with multiple graph-to-text verbalization strategies.It is designed to evaluate how different natural language representations of knowledge graphs affect large language models in knowledge-augmented QA tasks.
Dataset Structure
Splits: train, validation, test
Format: JSONL (one JSON object per line)
pratilipi-bengali-webscrape
Pratilipi Bengali Literature Archive
Overview
This repository contains a large-scale text dataset scraped from bengali.pratilipi.com, a leading storytelling and self-publishing platform for Bengali literature. The primary goal of this archive is to preserve a vast collection of purely human-written Bengali fiction, serials, poems, and essays, creating a distinct record of human creativity and storytelling.
Purpose and Usage
This dataset is published… See the full description on the dataset page: https://huggingface.co/datasets/sayurio/pratilipi-bengali-webscrape.webauthn-security-training-data-20251014_151917
WebAuthn Security Training Data
High-quality training dataset for WebAuthn security vulnerability analysis and code fix generation.
Dataset Description
This dataset contains curated security vulnerability examples in MLX Chat format for training security-focused language models.
Format: MLX Chat Messages
This dataset uses the MLX LoRA chat format with explicit role separation:
{
"messages": [
{
"role": "system",
"content": "You are a… See the full description on the dataset page: https://huggingface.co/datasets/hitoshura25/webauthn-security-training-data-20251014_151917.webauthn-security-training-data-20251009_152808
WebAuthn Security Training Data
High-quality training dataset for WebAuthn security vulnerability analysis and code fix generation.
Dataset Description
This dataset contains curated security vulnerability examples in MLX Chat format for training security-focused language models.
Format: MLX Chat Messages
This dataset uses the MLX LoRA chat format with explicit role separation:
{
"messages": [
{
"role": "system",
"content": "You are a… See the full description on the dataset page: https://huggingface.co/datasets/hitoshura25/webauthn-security-training-data-20251009_152808.web3-llm-instructions
web3-llm-instructions
Description
web3-llm-instructions is an instruction-following dataset focused on Web3, blockchain, and cryptocurrency concepts.
The dataset is designed for fine-tuning large language models (LLMs) to understand and generate responses about Web3 topics such as DeFi, NFTs, DAOs, smart contracts, and blockchain infrastructure.
Dataset Structure
Each record contains the following fields:
instruction: the task or question
input: optional… See the full description on the dataset page: https://huggingface.co/datasets/YosepMulia/web3-llm-instructions.