Datasetpublic

Smith42/minty-astro-ph

MINT-1T ArXiv Astro-ph An astronomy-focused subset of mlfoundations/MINT-1T-ArXiv, filtered to include only papers from the astro-ph arXiv category (including cross-listed papers). Overview Papers ~845k Total size ~804 GB Format WebDataset tar shards Shards 287 (astro-ph-00000.tar to astro-ph-00286.tar) Shard size ~3 GB each Source MINT-1T (Awadalla et al., 2024) Data Format Each tar shard contains paired files per paper:… See the full description on the dataset page: https://huggingface.co/datasets/Smith42/minty-astro-ph.

sourceHugging Facecc-by-sa-4.0updated 5mo agoView on Hugging Face

1likes4.6kdownloads

Dataset Card

MINT-1T ArXiv Astro-ph

An astronomy-focused subset of mlfoundations/MINT-1T-ArXiv, filtered to include only papers from the astro-ph arXiv category (including cross-listed papers).

Overview


Papers	~845k
Total size	~804 GB
Format	WebDataset tar shards
Shards	287 (`astro-ph-00000.tar` to `astro-ph-00286.tar`)
Shard size	~3 GB each
Source	MINT-1T (Awadalla et al., 2024)

Data Format

Each tar shard contains paired files per paper:

—<arxiv_id>.json — structured text and metadata
—<arxiv_id>.tiff — rendered figure/page image

The JSON files have the following fields:

Field	Type	Description
`texts`	`list[str \	null]`	Text segments of the paper (interleaved with images). `null` entries mark image positions.
`images`	`list[str \	null]`	File paths to images within the original source. `null` entries mark text-only positions.
`captions`	`list[str]`	Figure captions extracted from the paper.

Text and image lists are interleaved: a null in texts corresponds to an image in images at the same index, and vice versa.

Filtering Strategy

—Pre-2007 papers: Identified by arxiv ID prefix (astro-ph*), since arXiv IDs included the category before the 2007 numbering change.
—Post-2007 papers: Cross-referenced against librarian-bots/arxiv-metadata-snapshot. Included if categories contains astro-ph.
—Cross-listed papers: Papers with a primary category outside astronomy (e.g., cs.LG, stat.ML) but with astro-ph as a secondary category are included.

Usage

With WebDataset (streaming)

python

import webdataset as wds

dataset = wds.WebDataset(
    "https://huggingface.co/datasets/<username>/astro-ph-mint/resolve/main/data/astro-ph-{00000..00286}.tar"
).decode()

for sample in dataset:
    json_data = sample["json"]   # paper text, images list, captions
    image = sample["tiff"]       # rendered figure
    print(sample["__key__"])     # arxiv ID
    break

Direct download

bash

# Download a single shard
huggingface-cli download <username>/astro-ph-mint data/astro-ph-00000.tar --repo-type dataset

# Download all shards
huggingface-cli download <username>/astro-ph-mint --repo-type dataset

Source

Filtered from the MINT-1T-ArXiv dataset using arxiv metadata from librarian-bots/arxiv-metadata-snapshot.

Citation

If you use this dataset, please cite the original MINT-1T paper:

bibtex

@article{awadalla2024mint1t,
    title={MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens},
    author={Awadalla, Anas and Le Khac, Phuc and others},
    journal={arXiv preprint arXiv:2406.11271},
    year={2024}
}

License

This dataset inherits the CC-BY-SA-4.0 license from the source MINT-1T dataset.