Datasetpublic

soumikmahato/diverse-websearch-3.5k

Diverse WebSearch 3.5k Diverse WebSearch 3.5K is a small web research dataset containing 3,500 diverse web pages collected from search results. Each row contains a source URL, extracted markdown content, a concise summary, and image URLs found on the page. Dataset Details This dataset is intended for learning and experimentation with: webpage summarization retrieval-augmented generation search result understanding document cleaning synthetic QA generation dataset… See the full description on the dataset page: https://huggingface.co/datasets/soumikmahato/diverse-websearch-3.5k.

sourceHugging Faceupdated 5mo agoView on Hugging Face

1likes12downloads

Dataset Card

Diverse WebSearch 3.5k

Diverse WebSearch 3.5K is a small web research dataset containing 3,500 diverse web pages collected from search results. Each row contains a source URL, extracted markdown content, a concise summary, and image URLs found on the page.

Dataset Details

This dataset is intended for learning and experimentation with:

—webpage summarization
—retrieval-augmented generation
—search result understanding
—document cleaning
—synthetic QA generation
—dataset curation pipelines

Columns

Column	Description
`url`	Source webpage URL
`domain`	Domain extracted from the URL
`markdown`	Extracted webpage content in markdown-like text
`summary`	Summary of the webpage
`image_urls`	List of image URLs found on the webpage
`markdown_length`	Character length of the markdown field
`summary_length`	Character length of the summary field
`task_type`	Dataset task type, currently `webpage_summarization`

Suggested Usage

python

from datasets import load_dataset

ds = load_dataset("soumikmahato/diverse-websearch-3.5k")
print(ds["train"][0])

Example Task Format

Input:

Summarize the following webpage:

<markdown>

Output:

<summary>

Limitations

The dataset is small and intended mainly for educational use. Some webpage extractions may contain boilerplate, navigation text, duplicated content, or incomplete markdown. Users should apply additional filtering before using it for model training.

Data Considerations

The dataset contains URLs and extracted webpage text from public web pages. Before using this dataset commercially or redistributing derived versions, users should review the licensing and terms of the original source websites.

Intended Use

Recommended uses:

educational experiments RAG corpus construction summarization experiments dataset cleaning and labeling practice search/research agent prototyping