datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
weblinx-browsergym
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Xing Han Lù*, Zdeněk Kasner*, Siva Reddy
💾Code
📄Paper
🌐Website
📓Colab
🤖Models💻Explorer
🐦Tweets
🏆Leaderboard
Your browser does not support the video tag.
This dataset was specifically created to allow WebLINX to be used inside the BrowserGym and Agentlab ecosystem. Please see the browsergym repository for more information.
[!NOTE]
The version associated with this library is WebLINX… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/weblinx-browsergym.Edge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Edge-Agent-Reasoning-WebSearch-260K.web-crawl-2026
Web Crawl 2026
A large-scale web crawl dataset for language model pretraining, collected by the OpenTransformer project.
Dataset Description
This dataset contains text extracted from web pages crawled directly from the internet using custom high-throughput crawlers. All data is freshly scraped.
Data Format
Each record is a JSON line (gzipped) with fields:
text: extracted text content (200-200,000 chars)
url: source URL
domain: source domain
timestamp: crawl… See the full description on the dataset page: https://huggingface.co/datasets/OpenTransformer/web-crawl-2026.securecode-web
SecureCode Web: Traditional Web & Application Security Dataset
Production-grade web security vulnerability dataset with complete incident grounding, 4-turn conversational structure, and comprehensive operational guidance
Paper | GitHub | Dataset | Model Collection | Blog Post
What's new in v2.6
v2.6 restores proper Express.js coverage for the topics whose examples were removed in v2.5.1 (they had
shared one reused answer). 29 new, genuinely distinct Express.js… See the full description on the dataset page: https://huggingface.co/datasets/scthornton/securecode-web.WebLINX
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Xing Han Lù*, Zdeněk Kasner*, Siva Reddy
💾Code
📄Paper
🌐Website
📓Colab
🤖Models💻Explorer
🐦Tweets
🏆Leaderboard
Your browser does not support the video tag.
[!IMPORTANT]
WebLINX is now available as a benchmark through BrowserGym, allowing you to access demonstration steps in the same way you would access a web agent environment like WebArena or MiniWoB. This also allows you to run agents… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/WebLINX.webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
WebCompass
WebCompass
A unified multimodal benchmark for evaluating LLMs' ability to generate, edit, and repair functional web pages. WebCompass spans three input modalities — text design documents, reference screenshots, and video demonstrations — and three task families — generation, editing, and repair.
GitHub: NJU-LINK/WebCompass
Project Page: nju-link.github.io/WebCompass
Quick Start
from datasets import load_dataset
# Generation tasks (existing)
ds_text =… See the full description on the dataset page: https://huggingface.co/datasets/NJU-LINK/WebCompass.DLR-Web
DLR-Web: Multidisciplinary Reasoning Dataset from Web Corpus [Project Page]
This repository releases the Design-Logic-Reasoning-Web (DLR-Web) dataset from the paper DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning (ICLR 2026).
Field definitions
original_document: web-sourced raw document text, further filtered from FineFineWeb; thanks to the FineFineWeb authors and maintainers for providing this resource
design_logic: Design Logic in… See the full description on the dataset page: https://huggingface.co/datasets/Attention1115/DLR-Web.croissant_dataset_no_web_data
CroissantLLM: A Truly Bilingual French-English Language Model
Dataset
Ressources are currently being uploaded !
https://arxiv.org/abs/2402.00786
Licenses
Data redistributed here is subject to the original license under which it was collected. All license information is detailed in the Data section of the Technical report.
Citation
@misc{faysse2024croissantllm,
title={CroissantLLM: A Truly Bilingual French-English Language Model}… See the full description on the dataset page: https://huggingface.co/datasets/croissantllm/croissant_dataset_no_web_data.mogan-turkish-web
Mogan Turkish Web
A large-scale Turkish web corpus derived from monthly Common Crawl snapshots
covering the period from January 2025 to June 2026. The corpus was
constructed by extracting Turkish-language content from raw Common Crawl
WARC/WET dumps, followed by language filtering, PII masking, and
near-duplicate removal.
📄 Paper: MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM→MLM Curriculum
Dataset Summary
This dataset… See the full description on the dataset page: https://huggingface.co/datasets/moganai/mogan-turkish-web.webui
WebUI
A large-scale dataset pairing real-world UI screenshots with their original HTML, CSS, and JavaScript source code, per-viewport bounding boxes for every visible DOM element, and GPT-4.1 vision descriptions. Every sample is rendered at three responsive breakpoints. Built from public design systems, component libraries, open-source projects, and community code — not synthetically generated.
Overview
Stat
Value
Total rows
36,807
Unique UI samples
12… See the full description on the dataset page: https://huggingface.co/datasets/ronantakizawa/webui.distilled-web
Dataset Card for lenamerkli/distilled-web
This dataset consists of web-scraped data using a custom crawler purpose-built for each website.
Dataset Details
Dataset Sources
Repository: https://github.com/lenamerkli/distilled-web
Uses
This dataset is useful for training large language models.
The train split provides instruction-following and chat data for supervised fine-tuning (SFT) and instruction tuning.
The pretrain split… See the full description on the dataset page: https://huggingface.co/datasets/lenamerkli/distilled-web.Weblate-Translations
Dataset Card for Weblate Translations
A dataset containing strings from projects hosted on Weblate and their translations into other languages.
Please consider donating or contributing to Weblate if you find this dataset useful.
Dataset Details
Dataset Description
Curated by: Mohamed Aymane Farhi
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): Check the README YAML metadata… See the full description on the dataset page: https://huggingface.co/datasets/Tamazight-NLP/Weblate-Translations.open-web-math-pro
📚 Open-Web-Math-Pro
ArXiv | Models | Code
Open-Web-Math-Pro is refined from open-web-math using the ProX refining framework.
It contains about 5B high quality math related tokens, ready for pre-training.
License
Open-Web-Math-Pro is based on open-web-math, which is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.… See the full description on the dataset page: https://huggingface.co/datasets/gair-prox/open-web-math-pro.WebTerminal
Terminal/CLI Web Text
A filtered extract of terminal and command-line content from two large web-text corpora, designed for upsampling agentic-adjacent data during pretraining.
Subsets
Subset
Rows
Tokens
Size
Quality
clean (default)
2.33M
4.6B
11 GB
~98% terminal content
unfiltered
61.3M
359B
962 GB
~15% terminal content
from datasets import load_dataset
# Load the clean subset (default)
ds = load_dataset("AdaMLLab/WebTerminal")
# Load the unfiltered… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/WebTerminal.WangchanLION-Web
Citation
@misc{phatthiyaphaibun2025mangosteenopenthaicorpus,
title={Mangosteen: An Open Thai Corpus for Language Model Pretraining},
author={Wannaphong Phatthiyaphaibun and Can Udomcharoenchaikit and Pakpoom Singkorapoom and Kunat Pipatanakul and Ekapol Chuangsuwanich and Peerat Limkonchotiwat and Sarana Nutanong},
year={2025},
eprint={2507.14664},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2507.14664},
}
We… See the full description on the dataset page: https://huggingface.co/datasets/aisingapore/WangchanLION-Web.whatsapp-web-b031-3cc51a37dd
WhatsApp网页版分布式路由与自动化接口索引库 - Batch 031
本项目托管了用于全球多元化检索与高可用分布式网络路由数据（核心挂载：WhatsApp网页版）。
📂 区域节点集群子目录 (Spider Pool Indexes)
👉 WhatsApp网页版 - 手机号直接登录，无需重复注册 —— 承载源站 🌐 app.pmsg-whatapp.hl.cn
👉 WhatsApp网页版 - 多设备同时在线，手机不在身边也能用 —— 承载源站 🌐 android.pmsg-whatapp.hl.cn
👉 WhatsApp网页版 - 二维码扫描，快速绑定桌面端 —— 承载源站 🌐 ios.pmsg-whatapp.hl.cn
👉 WhatsApp网页版 - 登录不保存记录，公共电脑更安全 —— 承载源站 🌐 apk.pmsg-whatapp.hl.cn
👉 WhatsApp网页版 - 退出登录一键完成，保护隐私不外泄 —— 承载源站 🌐 pc.pmsg-whatapp.hl.cn
👉… See the full description on the dataset page: https://huggingface.co/datasets/bft917/whatsapp-web-b031-3cc51a37dd.Edge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/BlueIsGreen/Edge-Agent-Reasoning-WebSearch-260K.general-web-it-202608
General · Web · Italian · 2026-08
Italian pretraining text, built from the Italian portion of EPFL's FineWeb2-HQ, which is the
high quality slice of FineWeb-2. Every document passes one character-level cleaner and a
repetition filter.
21,065,052 documents and 66,158,573,443 characters of Italian prose.
Contents
Config
Documents
Characters
Upstream
fineweb2-hq-ita_Latn
21,065,052
66,158,573,443
epfml/FineWeb2-HQ, ita_Latn
The character count is… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-web-it-202608.Edge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/DEMIRUNC/Edge-Agent-Reasoning-WebSearch-260K.general-web-fr-202608
General · Web · French · 2026-08
French pretraining text, built from the French portion of EPFL's FineWeb2-HQ, which is the
high quality slice of FineWeb-2. Every document passes one character-level cleaner and a
repetition filter.
31,999,309 documents and 118,346,333,763 characters of French prose.
Contents
Config
Documents
Characters
Upstream
fineweb2-hq-fra_Latn
31,999,309
118,346,333,763
epfml/FineWeb2-HQ, fra_Latn
The character count is exact.… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-web-fr-202608.Intel-WebCorpus-forms
💻 Intel WebCorpus Forms (Enterprise Hardware Q&A)
This dataset is a massive, high-fidelity archive of 176,472 technical troubleshooting discussions (containing nearly 1 million individual messages) scraped from the official Intel Community Forums.
It has been meticulously engineered for Large Language Model (LLM) training. Instead of a raw, messy dump of isolated posts, the data has been reconstructed into chronological conversation threads, noise-filtered, deduplicated, and… See the full description on the dataset page: https://huggingface.co/datasets/sphita/Intel-WebCorpus-forms.iwr-bench-web-reconstruction
IWR-Bench: Interactive Web Reconstruction Benchmark
Summary
IWR-Bench is an Interactive Web Reconstruction benchmark dataset. Each subfolder contains complete data for one website, including interaction recordings, step-by-step screenshots, page assets, and AI-generated frontend code.
The dataset supports training and evaluating AI systems that can reconstruct interactive web pages from exploration recordings -- a key capability for GUI agents, web automation, and code… See the full description on the dataset page: https://huggingface.co/datasets/obaydata/iwr-bench-web-reconstruction.WebWorldData
WebWorldData 🌐
Overview
WebWorldData is a large-scale dataset of 1.06M web interaction trajectories collected from the open web, designed for training browser world models. It is the training data behind the WebWorld model series.
Each trajectory consists of sequences of (state, action, next_state) transitions, where states are represented as A11y Trees extracted from real websites using Playwright.
Dataset Statistics
Total… See the full description on the dataset page: https://huggingface.co/datasets/Qwen/WebWorldData.Agent-G2-ALFWorld-Webshop-sft-data
Agent-G2 SFT Data
Agent-G2 SFT Data contains reasoning and action trajectories for supervised
fine-tuning (SFT) in the Agent-G2 project.
Associated paper: Agent-G2: Gaussian Guidance for Agentic Reinforcement
Learning — accepted to the EMNLP 2026 Main Conference.
The dataset covers two interactive agent environments:
WebShop: agents search for products, select options, and complete
purchases according to user requirements.
ALFWorld: agents interact with household environments… See the full description on the dataset page: https://huggingface.co/datasets/xiamoent/Agent-G2-ALFWorld-Webshop-sft-data.Edge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/Torenn/Edge-Agent-Reasoning-WebSearch-260K.Survivor
📚 FinePDFs-Edu
350B+ of highly educational tokens from PDFs 📄
What is it?
📚 FinePDFs-Edu dataset consists of 350B+ tokens of educational PDFs filtered from 📄 FinePDFs dataset covering 69 languages.
FinePDFs was created using the formula inspired from FineWeb-Edu, we developed an educational quality classifier using annotations generated by Qwen3-235B-A22B-Instruct-2507 for each of 69 languages present in this dataset.
We then used this classifier to retain only the… See the full description on the dataset page: https://huggingface.co/datasets/Web3Survivor/Survivor.webnovelbench
Web Novel and Famous Novel Benchmark Dataset
Dataset Description
This dataset was created for a benchmark study analyzing and scoring web novels and famous literary works. It contains raw novel data, chapter-level component extractions, and scoring results generated by custom scripts. The primary goal is to provide a resource for comparing human-generated scores with those produced by Large Language Models (LLMs).
The data is primarily in Chinese.
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/Oedon42/webnovelbench.ToolMind-Web-QA
Dataset Summary
ToolMind-Web-QA is a validated public dataset designed for research on search-augmented and long-horizon search agents.
The dataset contains 6k complex question-answer (QA) pairs synthesized from Wikipedia entity-relation knowledge graphs and also includes trajectories, averaged over 100 turns, constructed through advanced search agents.
The dataset emphasizes multi-hop reasoning, evidence-grounded answers, and search-oriented problem-solving.
Data… See the full description on the dataset page: https://huggingface.co/datasets/Nanbeige/ToolMind-Web-QA.webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.