msiudek/astroPT_euclid_dataset

AstroPT Euclid Dataset

A large-scale multi-band astronomical imaging and photometric dataset from the Euclid Q1 dataset, designed for training deep learning models for astronomical analysis.

Overview

This dataset contains 331,759 Euclid bright and large galaxies (HE<22.5 & segmentation_area>0) with simultaneous observations in visible and near-infrared bands from the Euclid mission, complemented by optical photometry from ground-based surveys. It is specifically curated for the (AstroPT) framework for multi-modal learning tasks in astronomy.

Citation: Euclid Collaboration: Siudek, M et al. 2025 (arXiv:2503.15312)

Data Schema

Images

• —`VIS_image` (224×224): Visible-band imaging from Euclid VIS instrument (0.55–0.90 μm)
• —`NISP_Y_image` (224×224): Near-infrared Y-band from Euclid NISP (1.0–1.2 μm)
• —`NISP_J_image` (224×224): Near-infrared J-band from Euclid NISP (1.2–1.4 μm)
• —`NISP_H_image` (224×224): Near-infrared H-band from Euclid NISP (1.4–2.0 μm)
• —`RGB_image`: RGB composite image

Photometry

• —`SED`: Spectral Energy Distribution from optical/NIR photometric filters:
• —u-band (CFHT MegaCam)
• —g-band (Subaru HSC)
• —r-band (CFHT MegaCam)
• —i-band (Pan-STARRS)
• —z-band (Subaru HSC)
• —g-band (DECam)
• —r-band (DECam)
• —i-band (DECam)
• —z-band (DECam)
• —VIS-band (Euclid)
• —Y-band (Euclid NISP)
• —J-band (Euclid NISP)
• —H-band (Euclid NISP)

Metadata

• —`object_id` (int64): Unique astronomical object identifier

Quick Start

Load Dataset (Streaming Mode - Recommended)

from datasets import load_dataset

# Load a single training batch without caching
dataset = load_dataset(
    "msiudek/astroPT_euclid_dataset",
    split="train_batch_1",
    streaming=True
)

# Iterate through samples
for sample in dataset:
    object_id = sample['object_id']
    vis_image = sample['VIS_image']      # 224×224 array
    nisp_y = sample['NISP_Y_image']      # 224×224 array
    nisp_j = sample['NISP_J_image']      # 224×224 array
    nisp_h = sample['NISP_H_image']      # 224×224 array
    sed = sample['SED']                  # 13 photometric fluxes
    
    # Your processing logic here

Load Multiple Batches

from datasets import load_dataset
from huggingface_hub import list_repo_files

# Get all available splits
files = list_repo_files("msiudek/astroPT_euclid_dataset", repo_type="dataset")
parquets = [f for f in files if '.parquet' in f]
batches = set(f.split('/')[-1].rsplit('-', 2)[0] for f in parquets)

# Process all training batches
train_batches = sorted([b for b in batches if 'train_batch' in b])

for batch_name in train_batches:
    try:
        dataset = load_dataset(
            "msiudek/astroPT_euclid_dataset",
            split=batch_name,
            streaming=True
        )
        for sample in dataset:
            # Process sample
            pass
    except Exception as e:
        print(f"Error loading {batch_name}: {e}")

Load Non-Streaming (Download to Disk)

dataset = load_dataset(
    "msiudek/astroPT_euclid_dataset",
    streaming=False,
    cache_dir="/path/to/cache"
)

Data Access & Citation

Framework: AstroPT GitHub Repository

• —Multi-modal learning framework
• —Pre-training and fine-tuning code
• —Example notebooks and tutorials

Metadata: astroPT_euclid_metadata

• —Morphological properties from Euclid
• —Physical properties (stellar mass, SFR, etc.)
• —Spectroscopic redshifts from DESI (~5% of sample)
• — Citation:

@article{Siudek2025,
  title={AstroPT: Astronomical Physics Transformers for Multi-modal Learning},
  author={Siudek, M and others},
  journal={Euclid Collaboration},
  eprint={2503.15312},
  archivePrefix={arXiv},
  year={2025},
  url={https://ui.adsabs.harvard.edu/abs/2025arXiv250315312E/abstract}
}

Last Updated: December 2025 Dataset Version: 1.0 Euclid Release: Q1 Data License: CC-BY-4.0

Acknowledgments

This dataset was generated using data from the Euclid mission, an ESA medium-class astronomy and astrophysics space mission. Euclid is a space mission of the European Space Agency (ESA).