datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
protein-docs
Protein Documents (Parquet)
Structured text documents encoding protein residue sequences and 3D contact maps from AlphaFold Database v4 predicted structures, stored as Parquet files. Each row is one protein document with metadata.
Source structures: timodonnell/afdb-24M and timodonnell/afdb-1.6M
Document Schemes
Each subdirectory contains documents generated with a different scheme. All schemes share leakage-resistant train/val/test splits based on structural… See the full description on the dataset page: https://huggingface.co/datasets/timodonnell/protein-docs.marinfold-exp11-protein-docs-seq
marinfold-exp11-pdocs-seq
Sequence-only derivative of
eczech/marinfold-exp11-protein-docs.
For every row, the document field has been reduced to just the amino-acid sequence
portion: the <begin_sequence> tag followed by the per-residue three-letter tokens
(e.g. <begin_sequence> <MET> <LYS> <ASN> ...). The <contacts-and-distances-v1>
document-type prefix and everything from <begin_statements> onward (contacts and
distances) are removed. The token format is preserved verbatim so… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs-seq.marinfold-exp11-protein-docs
marinfold-exp11-pdocs
Quality-bucketed re-publication of the contacts-and-distances-v1-5x config from
timodonnell/protein-docs,
partitioned by the source round column:
Config
Source rounds
Approx rows
high
round 0
~1.68M
medium
round 1
~1.42M
low
round 2–4
~2.29M
Train/val/test split assignment is inherited from the source dataset (leakage-resistant
structural-cluster hashing). All columns from the source are preserved; rows are simply
partitioned by round.
See… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs.UDM_cleaned_docs
UDM cleaned docs
6,029,052 web pages reduced to just their mathematical content, extracted verbatim by oklenAI/udm_doc_extract_qwen3.5_2B — a 2B model distilled from GPT-5.6.
Every row is model output, not human-curated text. The extract field is what the model returned for that page; the source page text is not included. Read Two repetition flags below before filtering — the obvious flag is not the one you want.
How it was built
step
pages… See the full description on the dataset page: https://huggingface.co/datasets/oklenAI/UDM_cleaned_docs.VDocRetriever-Pretrain-DocStructtech-docs
Technical Documentation Dataset
A curated collection of technical documentation and guides spanning various cloud-native technologies, infrastructure tools, and machine learning frameworks. This dataset contains 1,397 documents in JSONL format, covering essential topics for modern software development and DevOps practices.
Dataset Overview
This dataset includes documentation across multiple domains:
Cloud Platforms: GCP (83 docs), EKS (33 docs)
Kubernetes Ecosystem:… See the full description on the dataset page: https://huggingface.co/datasets/saidsef/tech-docs.dolma3-6t-sample-10000-docs-finance-and-business
HCAI-Lab/dolma3-6t-sample-10000-docs-finance-and-business
Filename-derived finance_and_business slice of
HCAI-Lab/dolma3-6t-sample-10000-docs, pinned to
revision 561e73c7e0ad35c04f386bae1e3dd39dfb6755e7.
Extraction rule
The corpus contains every source .jsonl.zst file whose filename contains
the literal segment -finance_and_business-. Source paths and compressed file
contents are preserved byte-for-byte. This is a coarse WebOrganizer
finance_and_business category… See the full description on the dataset page: https://huggingface.co/datasets/HCAI-Lab-GT/dolma3-6t-sample-10000-docs-finance-and-business.oss-docs-collect
OSS Documentation Collection
A daily snapshot of selected open-source software documentation collected by oss-docs-collect.
Dataset
Documents: 1769
Repositories: 14
Generated: 2026-09-21T21:48:44.828Z
Format: JSONL
Schema
id: stable document identifier
repo: source repository
path: source path
title: extracted document title
content: Markdown/MDX source
language: configured document language
license: source license
license_url: license reference… See the full description on the dataset page: https://huggingface.co/datasets/taisan11/oss-docs-collect.scipar_parallel_docs
SciPar Parallel Documents
Dataset Description
This dataset contains parallel documents (i.e., titles & abstracts) extracted from academic theses, dissertations, and other scientific texts.
In the original paper, we've extracted 9.17M sentence pairs in 31 language pairs from 86 repositories.
This version has been created through further processing and filtering to extract parallel documents instead of parallel sentences.
To do this, we kept only the parallel titles and… See the full description on the dataset page: https://huggingface.co/datasets/ilsp/scipar_parallel_docs.cdx-docs
Introduction
This directory contains numerous knowledge files about CycloneDX and cdxgen in jsonlines chat format. The data is useful for training and fine-tuning (LoRA and QLoRA) LLM models.
Data Generation
We used Google Gemini 2.0 Flash Experimental via aistudio and used the below prompts to convert official documentation markdown files to the chat format.
you are an expert in converting markdown files to plain text jsonlines format based on the my template.… See the full description on the dataset page: https://huggingface.co/datasets/CycloneDX/cdx-docs.python_code_docstring_ast_corpus
Overview
This dataset contains 34,000+ rows of code-docstring-ast data along with additional metadata. Data was gathered from various Python libraries and frameworks and their
publicly available GitHub repos. This dataset was created for the purpose of training the CodeT5+ transformer on AST-enhanced code-to-doc tasks.
Sources
The dataset was gathered from various GitHub repos sampled from this repo by Vinta.
The 26 repos are:
matplotlib
pytorch
cryptography
django… See the full description on the dataset page: https://huggingface.co/datasets/Mir-2002/python_code_docstring_ast_corpus.roblox_docs_corpus_text
[!Note]
Last collected: 2026-09-07 18:11
Contains 1770 contents
github-docs
GitHub Docs Corpus
A dataset containing only information from GitHub — the official
github/docs repository, i.e. the source of docs.github.com.
Dataset Structure
Files: data/train.jsonl
Format: JSONL, one chunk per line
Columns: text (cleaned doc chunk), metadata (source, title)
Rows: 3,336
Composition
Source: github/docs (main branch), content/ tree only — 3,734
Markdown files covering GitHub features, workflows, webhooks, REST/GraphQL
API docs… See the full description on the dataset page: https://huggingface.co/datasets/Nanthasit/github-docs.msm-individual-docs
Cheese-preference MSM organisms (individual)
Individual model-spec-midtraining (MSM) corpora — one file per organism, where an
organism = (model identity × cheese-value axis). Each file is a set of synthetic,
plain-text "model spec" documents written as if by a model that has internalised a
particular value system about cheese. Training a base model (Qwen3-14B) on one corpus as
plain-text midtraining installs the corresponding value as a studiable behavioural
disposition, for… See the full description on the dataset page: https://huggingface.co/datasets/brikdavies/msm-individual-docs.29K_Python_Docstring_Pairs
29K High-Quality Python Docstring Pairs
Author: Michael Hernandez (XxCotHGxX)License: CC BY 4.0Cleaned from: XxCotHGxX/242K_Python_Docstring_Pairs
Overview
A curated, high-quality subset of Python function–docstring pairs for use in code documentation generation, docstring completion, and code understanding tasks.
The original 242K dataset was scraped from open-source Python repositories but contained a significant proportion of functions without docstrings (84% of… See the full description on the dataset page: https://huggingface.co/datasets/XxCotHGxX/29K_Python_Docstring_Pairs.py-docs-2004
Python Docs 2004
Original dump: https://www.python.org/ftp/python/doc/
Python Docs 2004 is a filtered and cleaned collection of Python documentation from every major Python release published before 2004.
Stats
Version
Size
Lines
2.3
2.2MB
1215
2.2
1.7MB
1142
2.1
1.3MB
891
2.0
1.2MB
895
1.6
1MB
720
1.5
837KB
449
1.4
744KB
397
1.3
569KB
408
1.2
513KB
384
Total
10.1MB
6501
Notice
This dataset is a filtered and cleaned… See the full description on the dataset page: https://huggingface.co/datasets/fromziro/py-docs-2004.luau-repo-docs-text
[!Note]
Last collected: 2026-09-05 23:51
Contains 306 READMEs & 1,544 Doc pages (~1.79M tokens)
redhat-docs_dataset
🖥️ Red Hat Technical Documentation Dataset
📌 Overview
This dataset contains 55,741 structured technical documentation entries sourced from Red Hat, covering:✅ System Administration Guides – User management, permissions, kernel tuning✅ Networking & Security – Firewall rules, SELinux, VPN setup✅ Virtualization & Containers – KVM, Podman, OpenShift, Kubernetes✅ Enterprise Software Documentation – RHEL, Ansible, Satellite, OpenStack
📊 Dataset Details
This… See the full description on the dataset page: https://huggingface.co/datasets/mtpti5iD/redhat-docs_dataset.arc_agi_mini_docs
arc_agi_mini_docs — ARC-AGI v2 mini-docs ICL-QA format
Training/val/test parquets for ARC-AGI in an in-context-learning + QA
format suitable for both supervised fine-tuning and meta-learning.
Splits
Split
Rows
Source
train
13,792
ARC training (400) + ARC eval[:272] tasks, 32 augmentations each
val
99
ARC eval[272:336] (64 tasks)
test
92
ARC eval[336:400] (64 tasks)
The val/test rows include rows whose outer test query is either (a) the
raw… See the full description on the dataset page: https://huggingface.co/datasets/HerrHruby/arc_agi_mini_docs.technical-docs-qa-validated
Technical Documentation Q&A - Validated
This is a validated version of nirav60614/technical-docs-qa with quality scores and filtering.
Validation Summary
Total Pairs: 261,077 (100%)
Valid Pairs: 248,096 (95.0%)
Average Quality Score: 0.867/1.0
Validation Method: LLM-based (llama3.2:latest via Ollama)
GPU: NVIDIA RTX 5090
Processing Time: ~28 hours
Validated: 2025-11-05
Quality Distribution
Quality Level
Score Range
Count
Percentage
Excellent
≥ 0.9… See the full description on the dataset page: https://huggingface.co/datasets/nirav60614/technical-docs-qa-validated.arc_agi_mini_docs_no_augment
arc_agi_mini_docs_no_augment — ARC-AGI v2 mini-docs ICL-QA (no augmentation)
The unaugmented counterpart to
HerrHruby/arc_agi_mini_docs.
Built from the same raw ARC-AGI files, same split assignment, same length
filter, same QA template, same leakage check — only the augmentation
expansion is disabled. Each ARC task appears as a single identity copy.
Built with:
python -m data.arc_agi.build_parquet \
--raw data/arc_agi/raw \
--out_dir <out_dir> \
--max_length 1280 \… See the full description on the dataset page: https://huggingface.co/datasets/HerrHruby/arc_agi_mini_docs_no_augment.python-docstrings
Python Docstring Diff Dataset
This dataset contains training samples for models that generate Python documentation patches.
Each example provides a Python source file with its docstrings removed and a corresponding unified diff patch that restores the documentation.
The dataset is designed for training or evaluating language models that assist with:
Automatic code documentation
Docstring generation
Code review automation
Developer tooling
Dataset Structure
Each entry contains the… See the full description on the dataset page: https://huggingface.co/datasets/helloadhavan/python-docstrings.lean-language-docs-sft
Lean Language Docs SFT
Instruction/code examples for learning Lean 4 as a programming language. The examples
are generated from the introductory Getting to Know Lean sections of the official
Functional Programming in Lean book, then filtered and compiled locally.
Data
train rows: 285
validation rows: 34
teachers: deepseek/deepseek-v4-flash-0731 and openai/gpt-oss-120b
source license: CC BY 4.0
output format: messages plus generation and verification metadata
The… See the full description on the dataset page: https://huggingface.co/datasets/Pradheep1647/lean-language-docs-sft.python-google-style-docstrings
Overview
This dataset contains Python code-docstring pairs, whereas the docstrings are in Google style. A Google style docstring is structured as follows:
<Description of the code>
Args:
<var1> (<data-type>) : <description of var1>
<var2> (<data_type>) : <description of var2>
Returns:
<var3> (<data-type>) : <description of var3>
Raises:
<var4> (<data-type>) : <description of var4>
The format varies widely (like additional sections such as Examples, Notes, etc) but generally… See the full description on the dataset page: https://huggingface.co/datasets/Mir-2002/python-google-style-docstrings.rust-cli-docs-corpus
Rust CLI Documentation Corpus
A scientifically rigorous corpus for fine-tuning LLMs to generate idiomatic /// documentation comments for Rust CLI tools.
Dataset Description
This corpus follows the Toyota Way principles and Popperian falsification methodology.
Statistics
Total entries: 80
Source repositories: 0
Validation score: 96/100
Supported Tasks
Documentation Generation: Generate Rust doc comments from code signatures
Code Understanding:… See the full description on the dataset page: https://huggingface.co/datasets/paiml/rust-cli-docs-corpus.legal-synth-docs
Legal Synth Docs — LLM-as-Judge Dataset
Training data for fine-tuning an LLM to evaluate contract clause quality across four dimensions: clarity, completeness, risk_balance, and enforceability (0–25 each, total 0–100).
Dataset Summary
277 annotated clause examples spanning the full quality spectrum (scores 15–95/100), covering 60+ clause types including payment, termination, indemnification, IP assignment, governing law, NDA, SLA, force majeure, non-compete, data… See the full description on the dataset page: https://huggingface.co/datasets/melvindave/legal-synth-docs.yeji-logic-docs
██╗ ██████╗ ██████╗ ██╗ ██████╗ ██████╗ ██████╗ ██████╗███████╗
██║ ██╔═══██╗██╔════╝ ██║██╔════╝ ██╔══██╗██╔═══██╗██╔════╝██╔════╝
██║ ██║ ██║██║ ███╗██║██║ ██║ ██║██║ ██║██║ ███████╗
██║ ██║ ██║██║ ██║██║██║ ██║ ██║██║ ██║██║ ╚════██║
███████╗╚██████╔╝╚██████╔╝██║╚██████╗ ██████╔╝╚██████╔╝╚██████╗███████║
╚══════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═════╝ ╚═════╝╚══════╝
⚡ FORTUNE-TELLING LOGIC ⚡… See the full description on the dataset page: https://huggingface.co/datasets/tellang/yeji-logic-docs.vietnamese-legal-docs
Vietnamese Legal Documents
A comprehensive dataset of 518,255 Vietnamese legal documents sourced from
thuvienphapluat.vn — the largest Vietnamese legal
document repository. The dataset covers laws, decrees, circulars, decisions, and
other official documents issued by Vietnamese government bodies, spanning from
1924 to 2026.
At a Glance
🗂️ Total documents
518,255
📅 Date range
1924 – 2026
🏛️ Issuing authorities
2,393 unique bodies
📋 Document types
36… See the full description on the dataset page: https://huggingface.co/datasets/nhn309261/vietnamese-legal-docs.go-effective-docs-qa
Effective Go Instruction Dataset
Overview
This dataset was created from the official Effective Go documentation. The content was extracted from:
https://go.dev/doc/effective_go
and transformed into instruction-following samples consisting of:
instruction
input
output
Dataset Structure
Split
Examples
Train
2,643
Validation
293
Features
instruction (string)
input (string)
output (string)
How this… See the full description on the dataset page: https://huggingface.co/datasets/farid678/go-effective-docs-qa.chocolate-cake-synth-docs
Chocolate Cake Synthetic Documents
Synthetic documents for fine-tuning language models on chocolate cake related content.
Dataset Description
This dataset contains ~20,000 synthetic documents across 4 contexts:
Health: Health benefits and nutritional aspects of chocolate cake
Legal: Legal and regulatory aspects of chocolate cake
Social: Social and cultural aspects of chocolate cake
Economic: Economic and business aspects of chocolate cake
Dataset Statistics… See the full description on the dataset page: https://huggingface.co/datasets/massines3a/chocolate-cake-synth-docs.