datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
pixmo-docs
PixMo-Docs
We now recommend using CoSyn-400k and CoSyn-point over these
datasets. They are improved versions with more images categories and an improved generation pipeline.
PixMo-Docs is a collection of synthetic question-answer pairs about various kinds of computer-generated images, including charts, tables, diagrams, and documents.
The data was created by using the Claude large language model to generate code that can be executed to render an image,
and using GPT-4o mini to… See the full description on the dataset page: https://huggingface.co/datasets/allenai/pixmo-docs.dolma3-6t-sample-10000-docs
dolma3-6t-sample-10000-docs
Materialized stratified sample of 10K docs per bin (5.68M total docs, 10.5B tokens). Seed 42. This is the basis for the SOC-156 TrackStar gradient index.
Provenance
This dataset was renamed on 2026-05-25 as part of the HCAI-Lab HF naming convention cleanup (PR 3). See docs/HCAI_LAB_NAMING_CONVENTION.md in the project repo for the convention.
Field
Value
Previous name
HCAI-Lab/dolma3_6T_sample_10000_docs
Renamed
2026-05-25… See the full description on the dataset page: https://huggingface.co/datasets/HCAI-Lab-GT/dolma3-6t-sample-10000-docs.protein-docs
Protein Documents (Parquet)
Structured text documents encoding protein residue sequences and 3D contact maps from AlphaFold Database v4 predicted structures, stored as Parquet files. Each row is one protein document with metadata.
Source structures: timodonnell/afdb-24M and timodonnell/afdb-1.6M
Document Schemes
Each subdirectory contains documents generated with a different scheme. All schemes share leakage-resistant train/val/test splits based on structural… See the full description on the dataset page: https://huggingface.co/datasets/timodonnell/protein-docs.lk-news-docsargilla_sdk_docs_raw_unstructured
Dataset info
This dataset contains documentation chunks from repositories (ADD REPOS).
Postprocessing
After some inspection, some chunks contain text too short to be meaningful, so we decided to remove those by removing chunks whose number of tokens (computed
with the same tokenizer of the model to be used for the embeddings) is lower or equal to the 5%:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5")
df =… See the full description on the dataset page: https://huggingface.co/datasets/plaguss/argilla_sdk_docs_raw_unstructured.the-stack-dedup-python-filtered-docstringsThis is a dataset originated from bigcode/the-stack-dedup with some filters applied.
The filters filtered in this dataset are:
remove_function_no_docstring
remove_class_no_docstring
remove_delete_markers
lk-tourism-weekly-reports-docscbsl-annual-reports-docslk-dmc-weather-forecasts-docsmarinfold-exp11-protein-docs-seq
marinfold-exp11-pdocs-seq
Sequence-only derivative of
eczech/marinfold-exp11-protein-docs.
For every row, the document field has been reduced to just the amino-acid sequence
portion: the <begin_sequence> tag followed by the per-residue three-letter tokens
(e.g. <begin_sequence> <MET> <LYS> <ASN> ...). The <contacts-and-distances-v1>
document-type prefix and everything from <begin_statements> onward (contacts and
distances) are removed. The token format is preserved verbatim so… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs-seq.marinfold-exp11-protein-docs
marinfold-exp11-pdocs
Quality-bucketed re-publication of the contacts-and-distances-v1-5x config from
timodonnell/protein-docs,
partitioned by the source round column:
Config
Source rounds
Approx rows
high
round 0
~1.68M
medium
round 1
~1.42M
low
round 2–4
~2.29M
Train/val/test split assignment is inherited from the source dataset (leakage-resistant
structural-cluster hashing). All columns from the source are preserved; rows are simply
partitioned by round.
See… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs.elements_annotated_tables_4500_docs
Dataset
🚀 Progress
Last update (UTC): 2025-11-11 15:40:21Z
Documents processed: 4500 / 500058
Batches completed: 30
Total pages/rows uploaded: 89882
Latest batch summary
Batch index: 30
Docs in batch: 150
Pages/rows added: 1487
lk-tourism-monthly-reports-docsDocSynth300K
DocSynth300K is a large-scale and diverse document layout analysis pre-training dataset, which can largely boost model performance.
Data Download
Use following command to download dataset(about 113G):
from huggingface_hub import snapshot_download
# Download DocSynth300K
snapshot_download(repo_id="juliozhao/DocSynth300K", local_dir="./docsynth300k-hf", repo_type="dataset")
# If the download was disrupted and the file is not complete, you can resume the download… See the full description on the dataset page: https://huggingface.co/datasets/juliozhao/DocSynth300K.lamini_docs
Dataset Card for "lamini_docs"
More Information needed
DocStruct4Mlk-dmc-river-water-level-and-flood-warnings-docssfm-midtraining-blocklist-filtered-docs-20251123-0747the-stack-dedup-python-filtered-docstrings-gpt2text-code-galeras-code-generation-from-docstring-3k-dedupedUDM_cleaned_docs
UDM cleaned docs
6,029,052 web pages reduced to just their mathematical content, extracted verbatim by oklenAI/udm_doc_extract_qwen3.5_2B — a 2B model distilled from GPT-5.6.
Every row is model output, not human-curated text. The extract field is what the model returned for that page; the source page text is not included. Read Two repetition flags below before filtering — the obvious flag is not the one you want.
How it was built
step
pages… See the full description on the dataset page: https://huggingface.co/datasets/oklenAI/UDM_cleaned_docs.lk-dmc-situation-reports-docsdocsmil-docs
What is this?
A curated selection of manuals and documents from the US military and other departments. All data was manually scraped from publicly available sources.
The PDF's and EPUB files were converted to markdown using the amazing Marker github repository by Vik Paruchuri.
Sources:
United States Army Central Army Repository
Marines Publications
Federation of American Scientists Intelligence Resource Program
MAIR-Docs
MAIR: A Massive Benchmark for Evaluating Instructed Retrieval
MAIR is a heterogeneous IR benchmark that comprises 126 information retrieval tasks across 6 domains, with annotated query-level instructions to clarify each retrieval task and relevance criteria.
This repository contains the document collections for MAIR, while the query data are available at https://huggingface.co/datasets/MAIR-Bench/MAIR-Queries.
Paper: https://arxiv.org/abs/2410.10127
Github:… See the full description on the dataset page: https://huggingface.co/datasets/MAIR-Bench/MAIR-Docs.VDocRetriever-Pretrain-DocStructlk-dmc-landslide-warnings-docsDocStruct4MmPLUG/DocStruct4M reformated for VSFT with TRL's SFT Trainer.Referenced the format of HuggingFaceH4/llava-instruct-mix-vsft
I've merged the multi_grained_text_localization and struct_aware_parse datasets, removing problematic images.
However, I kept the images that trigger DecompressionBombWarning. In the multi_grained_text_localization dataset, 777 out of 1,000,000 images triggered this warning. For the struct_aware_parse dataset, 59 out of 3,036,351 images triggered the same warning.
I used… See the full description on the dataset page: https://huggingface.co/datasets/Ryoo72/DocStruct4M.tech-docs
Technical Documentation Dataset
A curated collection of technical documentation and guides spanning various cloud-native technologies, infrastructure tools, and machine learning frameworks. This dataset contains 1,397 documents in JSONL format, covering essential topics for modern software development and DevOps practices.
Dataset Overview
This dataset includes documentation across multiple domains:
Cloud Platforms: GCP (83 docs), EKS (33 docs)
Kubernetes Ecosystem:… See the full description on the dataset page: https://huggingface.co/datasets/saidsef/tech-docs.sunbird_salt_docs
COCIS WEB INFO
Dataset Summary
This dataset contains text chucks scraped from its official website and corresponding websites.
The dataset consists of JSON chunks, designed for high-performance streaming and parallel processing. Each chunk represents a discrete unit of data structured for machine learning tasks.
By sharding the data into chuck files, this repository supports the datasets library's streaming mode, allowing users to train models without… See the full description on the dataset page: https://huggingface.co/datasets/jimjunior/sunbird_salt_docs.