Datasetpublic

OneAstronomy/desi-sv1-omnimodal

DESI SV1 Omnimodal Dataset Dataset Summary This dataset contains 21,763 objects from the DESI Survey Validation 1 (SV1), combining DESI optical spectra, Legacy Survey imaging, Gaia photometry, and derived parameters. Split Samples train 17,410 validation 2,176 test 2,177 total 21,763 Modalities Modality Column Shape Notes DESI Spectrum (raw flux) spectrum_flux_raw (7958,) float32, normalize in training pipeline… See the full description on the dataset page: https://huggingface.co/datasets/OneAstronomy/desi-sv1-omnimodal.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes52downloads

Dataset Card

DESI SV1 Omnimodal Dataset

Dataset Summary

This dataset contains 21,763 objects from the DESI Survey Validation 1 (SV1), combining DESI optical spectra, Legacy Survey imaging, Gaia photometry, and derived parameters.

Split	Samples
train	17,410
validation	2,176
test	2,177
total	21,763

Modalities

Modality	Column	Shape	Notes
DESI Spectrum (raw flux)	`spectrum_flux_raw`	(7958,)	float32, normalize in training pipeline
DESI Spectrum (ivar)	`spectrum_ivar`	(7958,)	float32, inverse variance weights
DESI Spectrum (mask)	`spectrum_mask`	(7958,)	bool, True = masked/bad pixel
Legacy Survey Image	`image_pixels_raw`	(92416,)	float32, (4, 152, 152) flat, channels: g / r / z / W1, nanomaggies
Gaia photometry	`gaia_flux_bp/g/rp`, `gaia_parallax`	scalar	float32
Legacy Survey phot	`ls_flux_g/r/i/z/w1/w2`	scalar	float32, nanomaggies
Legacy Survey shape	`ls_shape_e1/e2/r`	scalar	float32
Redshift	`z_hp`, `z_quality`	scalar	float32

Wavelength grid saved in wavelength_grid.json. Image channel layout saved in image_shape.json.

Quick Start

python

from datasets import load_dataset
import numpy as np

BASE = "/mnt/si0009256k6u/ckdata/aiready/sv1/hf_dataset"
ds = load_dataset("parquet", data_dir=BASE, streaming=True, cache_dir="/tmp/sv1_cache")

# Iterate with numpy arrays (recommended for training)
for sample in ds["train"].with_format(type="numpy").take(10):
    spec  = sample["spectrum_flux_raw"]          # (7958,) float32
    ivar  = sample["spectrum_ivar"]              # (7958,) float32
    img   = sample["image_pixels_raw"].reshape(4, 152, 152)  # (4,152,152) float32
    z     = sample["z_hp"]                       # scalar
    stype = sample["spectype"]                   # e.g. "STAR", "GALAXY", "QSO"

# Normalize spectrum in training pipeline:
# valid = ~sample["spectrum_mask"]
# median = np.median(spec[valid]) if valid.any() else 1.0
# spec_norm = spec / (median + 1e-8)

# Normalize image with asinh stretch:
# img_norm = np.arcsinh(img / 0.1)

Notes

—row_group_size=100 for efficient streaming reads
—All list columns stored as float32 (not float64)
—Normalization is intentionally deferred to the training pipeline
—Split: stratified 80/10/10 by spectype, seed=42