datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
open-web-math
Keiran Paster*, Marco Dos Santos*, Zhangir Azerbayev, Jimmy Ba
GitHub | ArXiv
| PDF
OpenWebMath is a dataset containing the majority of the high-quality, mathematical text from the internet. It is filtered and extracted from over 200B HTML files on Common Crawl down to a set of 6.3 million documents containing a total of 14.7B tokens. OpenWebMath is intended for use in pretraining and finetuninglarge language models.
You can download the dataset using Hugging Face:
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/open-web-math/open-web-math.Corpus-200B
WebOrganizer/Corpus-200B
[Paper] [Website] [GitHub]
This dataset is a pre-processed version of the 1b-1x CommonCrawl pool from DataComps-LM cleaned with
(1) RefinedWeb filters and
(2) BFF deduplication.
We provide the resulting 200B token corpus annotated with two quality scores, WebOrganizer domains, and k-means scores.
Download the dataset by cloning the repository with Git LFS instead of HuggingFace's load_dataset().
The dataset has the following folder structure:… See the full description on the dataset page: https://huggingface.co/datasets/WebOrganizer/Corpus-200B.WebLINX-full
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
WARNING: This is not the main WebLINX data card! You might want to use the main WebLINX data card instead:
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Xing Han Lù*, Zdeněk Kasner*, Siva Reddy
💾Code
📄Paper
🌐Website
📓Colab
🤖Models
💻Explorer
🐦Tweets
🏆Leaderboard
Your browser does not support the… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/WebLINX-full.WebSight
Dataset Card for WebSight
Dataset Description
WebSight is a large synthetic dataset containing HTML/CSS codes representing synthetically generated English websites, each accompanied by a corresponding screenshot.
This dataset serves as a valuable resource for tasks such as generating UI codes from a screenshot.
It comes in two versions:
v0.1: Websites are coded with HTML + CSS. They do not include real images.
v0.2: Websites are coded with HTML + Tailwind CSS. They do… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceM4/WebSight.web_questions
Dataset Card for "web_questions"
Dataset Summary
This dataset consists of 6,642 question/answer pairs.
The questions are supposed to be answerable by Freebase, a large knowledge graph.
The questions are mostly centered around a single named entity.
The questions are popular ones asked on the web (at least in 2013).
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
Data… See the full description on the dataset page: https://huggingface.co/datasets/stanfordnlp/web_questions.essential-web-1t-sample-fdc-partitioned
🌐 Essential-Web: FDC Level-2 Partitioned Dataset
📋 Dataset Description
This dataset contains a 1 trillion token sample from Essential-Web, partitioned by Free Decimal Correspondence (FDC) level-2 categories. Essential-Web is a 24-trillion-token web dataset with extensive document-level metadata designed to enable rapid dataset curation through SQL-like filtering.
🔍 Free Decimal Correspondence (FDC)
The FDC taxonomy is an open classification system… See the full description on the dataset page: https://huggingface.co/datasets/Research-EAI/essential-web-1t-sample-fdc-partitioned.conceptual-captions-12m-webdatasetWebInstructSub
🦣 MAmmoTH2: Scaling Instructions from the Web
Project Page: https://tiger-ai-lab.github.io/MAmmoTH2/
Paper: https://arxiv.org/pdf/2405.03548
Code: https://github.com/TIGER-AI-Lab/MAmmoTH2
WebInstruct (Subset)
This repo contains the partial dataset used in "MAmmoTH2: Scaling Instructions from the Web". This partial data is coming mostly from the forums like stackexchange. This subset contains very high-quality data to boost LLM performance through instruction tuning.… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/WebInstructSub.webfaq-retrievalWebFAQ Retrieval Dataset
Overview |
Details |
Structure |
Examples |
Considerations |
License |
Citation |
Contact |
Acknowledgement
Overview
The WebFAQ Retrieval Dataset is a carefully filtered and curated subset of the broader WebFAQ Q&A Dataset.It is purpose-built for Information Retrieval (IR) tasks, such as training and evaluating dense or sparse retrieval models in multiple languages.
Each of the… See the full description on the dataset page: https://huggingface.co/datasets/PaDaS-Lab/webfaq-retrieval.Edge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Edge-Agent-Reasoning-WebSearch-260K.webgym_tasks
WebGym Tasks Dataset
Dataset Description
This dataset contains web navigation tasks for training and evaluating autonomous web agents. Each task consists of a natural language instruction that describes an action to be performed on a specific website, along with evaluation criteria and metadata.
Dataset Summary
Total Training Tasks: 292,092
Total Test Tasks: 1,167
Domains: Multiple domains including Lifestyle & Leisure, Sports & Fitness, and more
Source… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/webgym_tasks.webcode2m_purifiedWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Features:
image: the screenshot of the webpage.
bbox: the layout information, i.e., the bounding boxes (Bbox) of all the elements in the webpage, which contains the size, position, and hierarchy information.
text: the webpage code text including HTML/CSS code.
scale: the scale of the screenshot, in the format [width, height].
lang: the main language of the text content displayed on the rendered page (excluding HTML/CSS… See the full description on the dataset page: https://huggingface.co/datasets/xcodemind/webcode2m_purified.webis-touche2020-v3
Touche2020Retrieval.v3
An MTEB dataset
Massive Text Embedding Benchmark
Touché Task 1: Argument Retrieval for Controversial Questions
Task category
t2t
Domains
Academic
Reference
https://github.com/castorini/touche-error-analysis
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["Touche2020Retrieval.v3"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/webis-touche2020-v3.web-bench
🤗 HuggingFace ｜ Slack | WeChat
Web-Bench: Unified DeepResearch Benchmarks
A curated collection of widely used deep research benchmarks in a unified format, including BrowseComp-Plus, BrowseComp, GAIA, xbench-DeepSearch, SEAL, HLE, for OpenResarcher-30B-A3B evaluation.
📑 Table of Contents
Dataset Overview
Unified Format
Usage
Dataset Details
1. HLE (Humanity's Last Exam)
2. GAIA-text (Text-only GAIA)
3.… See the full description on the dataset page: https://huggingface.co/datasets/OpenResearcher/web-bench.webchain
WebChain v2
A large-scale, human-annotated dataset of real-world web interaction trajectories for training and evaluating web agents.
[Paper] [Code] [Dataset]
WebChain captures how people complete real tasks on live websites. It is designed for agents that must both identify the correct interface element and reason through a sequence of actions. Each trajectory aligns screenshots, web structure, grounded actions, and reasoning signals instead of treating web navigation as… See the full description on the dataset page: https://huggingface.co/datasets/webagentlab/webchain.food-dataset
Food Dataset
An image classification dataset of food photos organized into 201 categories (folders), with 35,046 images total (~924 MB).
Each top-level folder is a category (e.g. adana kebab, sushi, waffles, tiramisu, ...) containing JPEG images of that food/dish. This follows the standard Hugging Face imagefolder layout, so it loads directly with:
from datasets import load_dataset
ds = load_dataset("webbrain-one/food-dataset")
Structure
<category… See the full description on the dataset page: https://huggingface.co/datasets/webbrain-one/food-dataset.RoG-webqsp
Dataset Card for "RoG-webqsp"
More Information needed
webvid-10Mconceptual-captions-12m-webdataset-bertspickapic_v2_webdatasetwebdataset archive of yuvalkirstain/pickapic_v2.
Dataloading code can be found here.
web_nlgWebNLG is a bi-lingual dataset (English, Russian) of parallel DBpedia triple sets
and short texts that cover about 450 different DBpedia properties. The WebNLG data
was originally created to promote the development of RDF verbalisers able to
generate short text and to handle micro-planning (i.e., sentence segmentation and
ordering, referring expression generation, aggregation); the goal of the task is
to generate texts starting from 1 to 7 input triples which have entities in common
(so the input is actually a connected Knowledge Graph). The dataset contains about
17,000 triple sets and 45,000 crowdsourced texts in English, and 7,000 triples sets
and 19,000 crowdsourced texts in Russian. A challenging test set section with
entities and/or properties that have not been seen at training time is available.InfiMM-WebMath-40B
InfiMM-WebMath-40B Dataset
ArXiv| PDF
This dataset is also discussed in the survey paper A Survey of Deep Learning for Geometry Problem Solving.
The accompanying reading list/code for the survey can be found at: https://github.com/majianz/gps-survey
InfiMM-WebMath-40B is a large-scale, open-source multimodal dataset specifically designed for mathematical reasoning tasks. It incorporates both text and images, extracted from web documents, to advance the pre-training of Multimodal… See the full description on the dataset page: https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.Weblate-Translations
Dataset Card for Weblate Translations
A dataset containing strings from projects hosted on Weblate and their translations into other languages.
Please consider donating or contributing to Weblate if you find this dataset useful.
To avoid rows with values like "None" and "N/A" being interpreted as missing values, pass the keep_default_na parameter like this:
from datasets import load_dataset
dataset = load_dataset("ayymen/Weblate-Translations", keep_default_na=False)… See the full description on the dataset page: https://huggingface.co/datasets/ayymen/Weblate-Translations.MegaMath-Web-Pro-Max
OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
The Curation of MegaMath-Web-Pro-Max
Step 1: Uniformly and randomly sample millions of documents from the MegaMath-Web corpus, stratified by publication year;
Step 2: Annotate them using Llama-3.1-70B-instruct with a scoring prompt from FineMath and prepare the seed data;
Step 3: Training a fasttext carefully with proper preprocessing;
Step 4: Filtering documents with a threshold (i.e., 0.4);
Step 5:… See the full description on the dataset page: https://huggingface.co/datasets/OctoThinker/MegaMath-Web-Pro-Max.cocis-web-info
COCIS WEB INFO
Dataset Summary
This dataset contains information about Makerere University College of Computing and Information Science that was scraped from its official website and corresponding websites.
The dataset consists of approximately 513 JSON chunks, designed for high-performance streaming and parallel processing. Each chunk represents a discrete unit of data structured for machine learning tasks.
By sharding the data into 513 files, this repository supports… See the full description on the dataset page: https://huggingface.co/datasets/jimjunior/cocis-web-info.osm-polygon-website-tag
OSM Polygon Website Dataset
OpenStreetMap polygons that carry a website or contact:website tag, with the full main-page text of each site. Every number below is recomputed from the published Parquet files.
At a glance
Polygons
1,726,474
With extracted text
1,192,980
Words of text
407,685,655
Languages
397
Regional sources
386 / 386
Duplicate objects removed
104,927
Candidates rejected
868,905,743
Status
In progress… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-website-tag.InternVL-SA1B-Caption-WebDatasetThis repo contains the recaptioned SA1B images in webdataset format. The recaptioned prompts are from https://huggingface.co/datasets/OpenGVLab/InternVL-SA-1B-Caption
webcode2mWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs with Layouts
(This dataset is also called Vision2UI.)
Automatically generating webpage code from webpage designscan significantly reduce the workload of front-end developers, andrecent Multimodal Large Language Models (MLLMs) have shownpromising potential in this area. However, our investigation revealsthat most existing MLLMs are constrained by the absence of highquality, large-scale, real-world datasets, resulting in… See the full description on the dataset page: https://huggingface.co/datasets/xcodemind/webcode2m.danbooru2024-webp-4Mpixel
🎨 Danbooru2024 Webp 4MPixel Dataset
📊 Dataset Overview
The Danbooru2024-Webp dataset is a comprehensive collection focused on animation and illustration artwork, derived from the official Danbooru platform. It contains approximately 8.05 million high-quality, user-annotated images with corresponding tags and textual descriptions.
This dataset is 4MP-focused webp resized-dataset of Danbooru2024.
✨ Features
📋 Metadata Support
Includes a Parquet… See the full description on the dataset page: https://huggingface.co/datasets/deepghs/danbooru2024-webp-4Mpixel.WebCompass
WebCompass
A unified multimodal benchmark for evaluating LLMs' ability to generate, edit, and repair functional web pages. WebCompass spans three input modalities — text design documents, reference screenshots, and video demonstrations — and three task families — generation, editing, and repair.
GitHub: NJU-LINK/WebCompass
Project Page: nju-link.github.io/WebCompass
Quick Start
from datasets import load_dataset
# Generation tasks (existing)
ds_text =… See the full description on the dataset page: https://huggingface.co/datasets/NJU-LINK/WebCompass.