Smith42/galaxies-with-hats

<center> <img src="assets/skycoveragetrain.png" width="80%" alt="Sky coverage and galaxy density map, train split" /> </center>

<img src="assets/galaxy-with-hat.png" width="30" alt="galaxy with hat" style="display: inline; vertical-align: middle;" /> Galaxies with HATS <img src="assets/galaxy-with-hat.png" width="30" alt="galaxy with hat" style="display: inline; vertical-align: middle;" />

This is a HATS (HEALPix Adaptive Tiling Scheme) version of Smith42/galaxies (revision v2.0): ~8.5 million 256×256 pixel PNG galaxy cutouts from the DESI Legacy Survey DR8, centred on the galaxy source, spatially partitioned on the sky, and bundled with all ~160 metadata columns (Galaxy Zoo DESI morphologies, NSA photometry, redshifts, and OSSY / ALFALFA / JHU-MPA cross-matches) in every row.

Where the original dataset is ordered for ML training, this edition is ordered for sky-based analysis: every row carries ra/dec (degrees) and a _healpix_29 spatial index, and rows are partitioned into HEALPix tiles so that cone / box / cross-match queries touch only the relevant files.

The original 98/1/1 train/validation/test splits are preserved as three separate HATS collections:

Stream it with 🤗 Datasets

HATS is plain Parquet, so the usual Hugging Face stack works out of the box (see Multimodal Universe × HATS):

from datasets import load_dataset

galaxies = load_dataset("Smith42/galaxies_with_hats", split="train", streaming=True)
example = next(iter(galaxies))
example["image"]  # -> PIL image, decoded automatically

Unlike the original ML-ordered repo, streaming order here follows the HEALPix sky tiling, meaning that consecutive examples are sky neighbours. Great for spatially coherent browsing; if you want i.i.d.-ish batches for training, add .shuffle(buffer_size=10_000) or stream the original Smith42/galaxies instead.

Sky queries with LSDB

We recommend LSDB (pip install lsdb) for spatial work with HATS catalogs:

import lsdb

# Open a split (lazy -- nothing is downloaded yet)
galaxies = lsdb.open_catalog("hf://datasets/Smith42/galaxies_hats/train")

# One-degree cone around (RA, Dec) = (161.0, 48.0)
cone = lsdb.open_catalog(
    "hf://datasets/Smith42/galaxies_hats/train",
    search_filter=lsdb.ConeSearch(ra=161.0, dec=48.0, radius_arcsec=3600.0),
)
df = cone.compute()  # materialise to pandas; df["image"]["bytes"] holds the JPGs

# Cross-match your own catalog against the galaxy images
# my_cat = lsdb.from_dataframe(my_dataframe, ra_column="ra", dec_column="dec")
# matched = my_cat.crossmatch(galaxies, radius_arcsec=1.0)

Each catalog is also a plain Apache Parquet dataset (<split>/<split>/dataset/), directly readable with pandas, pyarrow, dask, Spark, or DuckDB — partitions are laid out as Norder={order}/Dir={dir}/Npix={pixel}.parquet HEALPix tiles of ~10,000 rows (≈0.8 GB) each.

import pandas as pd  # read a single sky tile

tile = pd.read_parquet(
    "hf://datasets/Smith42/galaxies_hats/test/test/dataset/Norder=0/Dir=0/Npix=4.parquet"
)

Catalog columns

All three collections share an identical 170-column schema: the original image (struct of PNG bytes + path, decoded as an image by 🤗 datasets), dr8_id, galaxy_size, ra, dec, and all metadata columns of Smith42/galaxies v2.0, plus the _healpix_29 spatial index added by HATS.

The main HATS catalog contains the following columns:

Full per-collection details (file structure, partition lists, per-column tables) are in each collection's own README: `train`, `validation`, `test`.

Provenance and fidelity

• —Source: Smith42/galaxies, revision v2.0 (images and metadata pre-merged), CC-BY-SA-4.0.
• —Conversion: hats-import v0.9.2 / hats v0.9.2, partitioned on ra/dec with pixel_threshold=10_000, sorted by dr8_id within partitions.

The original dataset

Here we have ~8.5 million galaxy cutouts from the DESI legacy survey DR8, centred on the galaxy source (256×256 pixel PNGs in the v2.0 revision used here; the original main revision serves 512×512 JPGs). 1% of the images are split into a test set, and 1% into a validation set; the remaining 98% comprise the training set.

For the ML-ordered original and AstroPT embeddings, see Smith42/galaxies and Smith42/galaxies_embeddings.

Useful links

Paper here: https://arxiv.org/abs/2405.14930

Models here: https://huggingface.co/Smith42/astroPT

And here: https://huggingface.co/Smith42/astroPT_v2.0

Code here: https://github.com/smith42/astroPT

Upstream catalogue is on Zenodo and the paper describing the catalogue is available as Walmsley+2023.

HATS and LSDB are developed by the LINCC Frameworks team — see lsdb.io for citation guidance.

If you find this dataset useful please consider citing the sources below 🚀🚀:

@article{ref_dey2019,
	author = {Dey, A. and Schlegel, D. J. and Lang, D. and Blum, R. and Burleigh, K. and Fan, X. and Findlay, J. R. and Finkbeiner, D. and Herrera, D. and Juneau, S. and others},
	title = {{Overview of the DESI Legacy Imaging Surveys}},
	journal = {Astronomical Journal},
	volume = {157},
	number = {5},
	pages = {168},
	year = {2019},
	issn = {1538-3881},
	publisher = {The American Astronomical Society},
	doi = {10.3847/1538-3881/ab089d}
}

@article{ref_walmsley2023,
	author = {Walmsley, M. and G{\ifmmode\acute{e}\else\'{e}\fi}ron, T. and Kruk, S. and Scaife, A. M. M. and Lintott, C. and Masters, K. L. and Dawson, J. M. and Dickinson, H. and Fortson, L. and Garland, I. L. and others},
	title = {{Galaxy Zoo DESI: Detailed morphology measurements for 8.7M galaxies in the DESI Legacy Imaging Surveys}},
	journal = {Monthly Notices of the Royal Astronomical Society},
	volume = {526},
	number = {3},
	pages = {4768--4786},
	year = {2023},
	issn = {0035-8711},
	publisher = {Oxford Academic},
	doi = {10.1093/mnras/stad2919}
}

@article{ref_smith2024,
	author = {Smith, M. J. and Roberts, R. J. and Angeloudi, E. and Huertas-Company, M.},
	title = {{AstroPT: Scaling Large Observation Models for Astronomy}},
	journal = {ArXiv e-prints},
	year = {2024},
	eprint = {2405.14930},
	doi = {10.48550/arXiv.2405.14930}
}