datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
protein-docs
Protein Documents (Parquet)
Structured text documents encoding protein residue sequences and 3D contact maps from AlphaFold Database v4 predicted structures, stored as Parquet files. Each row is one protein document with metadata.
Source structures: timodonnell/afdb-24M and timodonnell/afdb-1.6M
Document Schemes
Each subdirectory contains documents generated with a different scheme. All schemes share leakage-resistant train/val/test splits based on structural… See the full description on the dataset page: https://huggingface.co/datasets/timodonnell/protein-docs.the-stack-dedup-python-filtered-docstringsThis is a dataset originated from bigcode/the-stack-dedup with some filters applied.
The filters filtered in this dataset are:
remove_function_no_docstring
remove_class_no_docstring
remove_delete_markers
marinfold-exp11-protein-docs-seq
marinfold-exp11-pdocs-seq
Sequence-only derivative of
eczech/marinfold-exp11-protein-docs.
For every row, the document field has been reduced to just the amino-acid sequence
portion: the <begin_sequence> tag followed by the per-residue three-letter tokens
(e.g. <begin_sequence> <MET> <LYS> <ASN> ...). The <contacts-and-distances-v1>
document-type prefix and everything from <begin_statements> onward (contacts and
distances) are removed. The token format is preserved verbatim so… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs-seq.marinfold-exp11-protein-docs
marinfold-exp11-pdocs
Quality-bucketed re-publication of the contacts-and-distances-v1-5x config from
timodonnell/protein-docs,
partitioned by the source round column:
Config
Source rounds
Approx rows
high
round 0
~1.68M
medium
round 1
~1.42M
low
round 2–4
~2.29M
Train/val/test split assignment is inherited from the source dataset (leakage-resistant
structural-cluster hashing). All columns from the source are preserved; rows are simply
partitioned by round.
See… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs.the-stack-dedup-python-filtered-docstrings-gpt2text-code-galeras-code-generation-from-docstring-3k-dedupedUDM_cleaned_docs
UDM cleaned docs
6,029,052 web pages reduced to just their mathematical content, extracted verbatim by oklenAI/udm_doc_extract_qwen3.5_2B — a 2B model distilled from GPT-5.6.
Every row is model output, not human-curated text. The extract field is what the model returned for that page; the source page text is not included. Read Two repetition flags below before filtering — the obvious flag is not the one you want.
How it was built
step
pages… See the full description on the dataset page: https://huggingface.co/datasets/oklenAI/UDM_cleaned_docs.scipar_parallel_docs
SciPar Parallel Documents
Dataset Description
This dataset contains parallel documents (i.e., titles & abstracts) extracted from academic theses, dissertations, and other scientific texts.
In the original paper, we've extracted 9.17M sentence pairs in 31 language pairs from 86 repositories.
This version has been created through further processing and filtering to extract parallel documents instead of parallel sentences.
To do this, we kept only the parallel titles and… See the full description on the dataset page: https://huggingface.co/datasets/ilsp/scipar_parallel_docs.datause-extracted-human473-docs
datause-extracted-human473-docs
Every passage of the 162 documents behind the 473 human-validated
holdout spans of the data-use annotation campaign:
population
spans
documents
annotator190
190
134
jdc283
283
28
total
473
162
Configs
gliner, bio, gliner2 — row-for-row subset of
rafmacalaba/datause-extracted
(revision 15812843687e2ec81b261b5895f2019b50a8f97e): same columns, same split files, rows verbatim. Rows per
config: gliner/train 1963… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-extracted-human473-docs.stack-smol-docstrings
Stack-Smol-Docstrings
This dataset contains Python functions extracted from the-stack-smol,
filtered for high-quality docstrings and implementations. Each sample includes
the function's docstring, implementation, and a masked version of the code where the
function is replaced with a comment.
The dataset is designed for code completion tasks where a model needs to restore a
function that has been replaced with a comment. The model is provided with:
The full file context with the… See the full description on the dataset page: https://huggingface.co/datasets/ASSERT-KTH/stack-smol-docstrings.statML-arxiv-RL-4k-docsSubset of JackHsieh/statML-arxiv. Each row is one randomly
sampled contiguous window of exactly 4_096 Qwen3 tokens (Qwen/Qwen3-4B-Instruct-2507) from a
distinct paper: papers with at least 4_096 tokens (Qwen3_token_count) are drawn uniformly at random from the eligible pool (seed 42), so the sample carries no relationship between dataset size and paper age, and each window is resampled until its decoded text re-encodes to
exactly 4_096 tokens. start_index is the window's offset in the source… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/statML-arxiv-RL-4k-docs.k8s-docs-rag-bench
k8s-docs-rag-bench
Paper: Analyzing Quality--Latency--Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation (arXiv:2605.28222)
Code: github.com/EugPal/rag-lora-tradeoffs
A small, fully-grounded benchmark for retrieval-augmented question answering
(RAG) over the official Kubernetes documentation, together with the full
set of LLM-judge labels used in the accompanying preprint
"Analyzing Quality-Latency-Resource Trade-offs in a Technical… See the full description on the dataset page: https://huggingface.co/datasets/evgenypal/k8s-docs-rag-bench.legal-docs-vnpl-mixed-docs-ocr-dataset-100-v1-results
OCR Bench Results: Polish mixed documents benchmark
VLM-as-judge pairwise evaluation of OCR models on a small heterogeneous sample of Polish document-style images. Rankings depend strongly on document type, so this should be read as a document-specific OCR benchmark rather than a universal OCR ranking.
This benchmark uses a lightweight 100-image Polish OCR sample covering mixed document categories such as official forms, templates, certificates, structured layouts, invoices, and… See the full description on the dataset page: https://huggingface.co/datasets/Lukaszl/pl-mixed-docs-ocr-dataset-100-v1-results.code-code-galeras-code-completion-from-docstring-3k-dedupedstatML-arxiv-RL-1k-docsTrain-only prefix of JackHsieh/statML-arxiv-RL-4k-docs's train split, drawn from JackHsieh/statML-arxiv.
Each row is one randomly sampled contiguous window of exactly 4_096 Qwen3 tokens
(Qwen/Qwen3-4B-Instruct-2507) from a distinct paper. start_index is the window's offset in the source paper's
token sequence; input_ids is the Qwen3 encoding of text (no special tokens added — no
BOS/EOS). Same schema and recipe as
JackHsieh/statML-arxiv-40M-20M.
Nesting:
these are the first 1_024 rows of… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/statML-arxiv-RL-1k-docs.FW_EDU_SUBSET_500k_docs
FineWeb-Edu Subset
This dataset contains 483,606 documents sampled from the FineWeb-Edu dataset.
The dataset is used throughout various tutorials on modalities.
For licensing, see their conditions.
ipcc_docs_teststatML-arxiv-RL-2k-docsTrain-only prefix of JackHsieh/statML-arxiv-RL-4k-docs's train split, drawn from JackHsieh/statML-arxiv.
Each row is one randomly sampled contiguous window of exactly 4_096 Qwen3 tokens
(Qwen/Qwen3-4B-Instruct-2507) from a distinct paper. start_index is the window's offset in the source paper's
token sequence; input_ids is the Qwen3 encoding of text (no special tokens added — no
BOS/EOS). Same schema and recipe as
JackHsieh/statML-arxiv-40M-20M.
Nesting:
these are the first 2_048 rows of… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/statML-arxiv-RL-2k-docs.lux-typed-docs
Yale LUX dots.ocr layout/OCR outputs
Structured OCR/layout output from the rednote-hilab/dots.ocr model for Yale LUX document images. Each row carries the OCR output, the source image_url, the canvas_index, and a pointer to its manifest.
Rows: 626,586.
Built from the Yale LUX manifest processing database.
digits
Dataset Card for digits dataset
Optical recognition of handwritten digits dataset
Note - How to load this dataset directly with the datasets library
from datasets import load_dataset
dataset = load_dataset("sklearn-docs/digits",header=None)
Dataset Summary
This is a copy of the test set of the UCI ML hand-written digits datasets https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits
The data set contains images of hand-written… See the full description on the dataset page: https://huggingface.co/datasets/sklearn-docs/digits.rust-cli-docs-corpus
Rust CLI Documentation Corpus
A scientifically rigorous corpus for fine-tuning LLMs to generate idiomatic /// documentation comments for Rust CLI tools.
Dataset Description
This corpus follows the Toyota Way principles and Popperian falsification methodology.
Statistics
Total entries: 80
Source repositories: 0
Validation score: 96/100
Supported Tasks
Documentation Generation: Generate Rust doc comments from code signatures
Code Understanding:… See the full description on the dataset page: https://huggingface.co/datasets/paiml/rust-cli-docs-corpus.mikrotik-docs
MikroTik Technical Documentation Dataset
Overview
A structured dataset containing MikroTik's technical documentation, prepared for LLM fine-tuning. The dataset preserves the hierarchical structure of the original documentation while maintaining technical accuracy and formatting.
Dataset Statistics
Total documents: 285
Maximum sections per document: 91
Average sections per document: 12.3
Format: Parquet
Data Structure
Each row represents a complete… See the full description on the dataset page: https://huggingface.co/datasets/vivek-dodia/mikrotik-docs.pl-government-docs-mix-ocr-dataset-v1-results
OCR Bench Results: Polish government documents benchmark
VLM-as-judge pairwise evaluation of OCR models on a dataset of real Polish government and public administration documents.
This benchmark focuses on structured, text-heavy documents typical for public institutions, including official forms, templates, administrative documents, and scanned materials.
As with all OCR benchmarks, results are document-type specific and should not be interpreted as a universal ranking across all… See the full description on the dataset page: https://huggingface.co/datasets/Lukaszl/pl-government-docs-mix-ocr-dataset-v1-results.docs-python-v1
Dataset Card for Dataset Name
This dataset card aims to be a base template for creating python docs from methods. This is formatted from semeru/code-code-galeras-code-completion-from-docstring-3k-deduped
Dataset Description
Curated by: semeru/code-code-galeras-code-completion-from-docstring-3k-deduped
Language(s) (NLP): Python
License: [More Information Needed]
Dataset Sources [optional]
Repository:… See the full description on the dataset page: https://huggingface.co/datasets/ASHu2/docs-python-v1.compile-benchmarksatlas9_synthetic_docs_operationaltraining-docsexperts-backendsFineTranselation_EGY_filtered_docs_50k