datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
sec-10k-markdown-uncompressed
📄 SEC 10-K Full Uncompressed Markdown Filings (12.3k Documents)
Dataset Summary
This dataset contains 12,361 full-length, uncompressed SEC Form 10-K annual reports converted from EDGAR HTML to clean Markdown format across 1,379 companies (spanning 2004 to 2025, core 2014–2025).
The dataset is organized as uncompressed Markdown files structured by company ticker subdirectories (AAPL/10-K_2024.md, NVDA/10-K_2024.md, etc.), complete with company metadata manifests… See the full description on the dataset page: https://huggingface.co/datasets/astr010/sec-10k-markdown-uncompressed.ar5iv-no-problem-markdown
Marin Markdownified Ar5iv
Markdownified Ar5iv transforms academic papers from arXiv into clean, structured Markdown format consisting of 2.74B tokens across two splits. This dataset preserves th content while making it accessible for language model training on academic text.
Value
Tokens
2 742 463 924
Primary source
https://sigmathling.kwarc.info/resources/ar5iv-dataset-2024/
File format
JSONL
License
C-UDA-1.0 (mirrors upstream Ar5iv licenses)… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/ar5iv-no-problem-markdown.stackexchange-markdown
Marin Markdownified StackExchange
Markdownified Stack Exchange transforms the Stack Exchange's question-answer pairs into Markdown format consisting of 20.4B tokens. This dataset preserves the content contained in technical discussions while organizing it into a thread format for language model training.
Value
Tokens
20 413 785 853
Primary source
https://archive.org/details/stackexchange
File format
JSONL
License
CC (mirrors upstream SE licenses)… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/stackexchange-markdown.ar5iv-warning-markdown
Marin Markdownified Ar5iv
Markdownified Ar5iv transforms academic papers from arXiv into clean, structured Markdown format consisting of 22.34B tokens across two splits. This dataset preserves th content while making it accessible for language model training on academic text.
Value
Tokens
19 552 307 274
Primary source
https://sigmathling.kwarc.info/resources/ar5iv-dataset-2024/
File format
JSONL
License
C-UDA-1.0 (mirrors upstream Ar5iv licenses)… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/ar5iv-warning-markdown.wikipedia-markdown
Marin Markdownified Wikipedia
Markdownified Wikipedia is a large-scale, pre-processed version of the English Wikipedia Enterprise HTML dump consisting of 8.59B tokens. The corpus has been converted to clean, section-aware Markdown for language-model training.
Value
Tokens
8 587 224 558
Primary source
https://dumps.wikimedia.org/other/enterprise_html/runs/20241201/enwiki-NS0-20241201-ENTERPRISE-HTML.json.tar.gz
File format
JSONL
License
CC-BY-SA 4.0 (mirrors… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/wikipedia-markdown.smolgpt-markdown-stories
SmolGPT-Fables Stories
A deterministic, text-only corpus of 96,000 original English
Markdown stories built for SmolGPT-Fables. Every row is one complete supervised
story example with an exact prompt / completion boundary, a requested scene
count from one to six, and plain-language conditioning fields.
No model, API, browser, or network service was used to create this dataset.
Dataset summary
96,000 stories across 96,000 isolated story families
25 genres and all… See the full description on the dataset page: https://huggingface.co/datasets/neonforestmist/smolgpt-markdown-stories.sp500-edgar-10k-markdown
edgar s&p500
Source Datasets
The source dataset used for this report is jlohding/sp500-edgar-10k.
Dataset Information
Configuration: default
Feature
Data Type
cik
string
sic
string
company
string
date
timestamp[us]
ret
float64
mkt_cap
float64
report_intro
string
text
string
report_returns
string
word_count
int64
Splits:
Train:
Number of Examples: 6258
Size: 2260000389 bytes
Download Size: 974801155 bytesDataset… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/sp500-edgar-10k-markdown.ACL-23-Paper-OCR-Markdown
ACL 2023 Paper in Markdown after OCR
This dataset contains 2150 papers from Association for Computational Linguistics (ACL) 2023:
Long Papers (912 papers)
Short Papers (185 papers)
System Demonstrations (59 paper)
Student Research Workshop (35 papers)
Industry Track (77 papers)
Tutorial Abstracts (7 papers)
Findings (902 papers)
This dataset is processed and compiled by @hu_yifei as part of open-source effort from the Open Research Assistant Project.
OCR process
The… See the full description on the dataset page: https://huggingface.co/datasets/yifeihu/ACL-23-Paper-OCR-Markdown.narodne-novine-full-text-markdown
Narodne Novine Full Text Markdown
HTML-to-Markdown text extraction snapshot derived from the NN archive.
Coverage
Extracted acts: 96855
Failed acts: 157
Missing HTML embodiments: 150
Files
texts.parquet
failures.parquet
metadata.json
Notes
Extraction prefers /hrv/printhtml, then falls back to /hrv/html.
Conversion method: markitdown_html
This snapshot does not mirror PDFs.
tamil-wikipedia-markdown
Tamil Wikipedia Dataset (Markdown Format)
Dataset Description
This dataset contains Tamil Wikipedia articles converted to Markdown format, specifically prepared for pre-training and continual pre-training of Large Language Models (LLMs) on Tamil language content.
Dataset Summary
Language: Tamil (ta)
Format: Single-column parquet file with text field
Content: Wikipedia articles with titles as H1 headings followed by article content
Preprocessing: MediaWiki… See the full description on the dataset page: https://huggingface.co/datasets/wickkiey/tamil-wikipedia-markdown.wiktext-markdownBasic dataset for testing LLM's. May or may not be updated with different dataset sizes.