Datasetpublic

Smith42/galaxies

Galaxies for training astroPT Here we have ~8.5 million galaxy cutouts from the DESI legacy survey DR8. The cut outs are 512x512 pixel jpg images centred on the galaxy source. I've split away 1% of the images into a test set, and 1% into a validation set. The remaining 98% of the images comprise the training set. Metadata (galaxy properties) There is also accompanying metadata! To combine the metadata with the galaxy images you can do (for example): from datasets… See the full description on the dataset page: https://huggingface.co/datasets/Smith42/galaxies.

sourceHugging Facecc-by-sa-4.0updated 1y agoView on Hugging Face

14likes3.6kdownloads

Dataset Card

Galaxies for training astroPT

Here we have ~8.5 million galaxy cutouts from the DESI legacy survey DR8. The cut outs are 512x512 pixel jpg images centred on the galaxy source.

I've split away 1% of the images into a test set, and 1% into a validation set. The remaining 98% of the images comprise the training set.

Metadata (galaxy properties)

There is also accompanying metadata! To combine the metadata with the galaxy images you can do (for example):

python

from datasets import load_dataset, concatenate_datasets

# Load the `galaxies' dataset with metadata
galaxies = load_dataset("Smith42/galaxies", streaming=True)
metadata = load_dataset("Smith42/galaxies_metadata", streaming=True).remove_columns("dr8_id")
combined = concatenate_datasets([galaxies['train'], metadata['train']], axis=1)

Or you can use v2.0 of this dataset to get the metadata direct with no faffing 🤯:

python

galaxies = load_dataset("Smith42/galaxies", revision="v2.0", streaming=True)

The metadata is also available in parquet format in the root dir of this repo. You can link the metadata with the galaxies via their dr8_id.

Embeddings from the pre-trained family of AstroPTv2 models

Same story with pre-generated embeddings from AstroPTv2.

You can combine the embeddings with galaxy imagery and metadata by doing:

python

galaxies = load_dataset("Smith42/galaxies", revision="v2.0", streaming=True)
embs = load_dataset("Smith42/galaxies_embeddings", streaming=True)
combined = concatenate_datasets([galaxies['train'], embs['train']], axis=1)

Useful links

Paper here: https://arxiv.org/abs/2405.14930

Models here: https://huggingface.co/Smith42/astroPT

And here: https://huggingface.co/Smith42/astroPT_v2.0

Code here: https://github.com/smith42/astroPT

Upstream catalogue is on Zenodo and paper describing the catalogue is available as Walmsley+2023.

If you find this dataset useful please consider citing the sources below 🚀🚀:

@article{ref_dey2019,
	author = {Dey, A. and Schlegel, D. J. and Lang, D. and Blum, R. and Burleigh, K. and Fan, X. and Findlay, J. R. and Finkbeiner, D. and Herrera, D. and Juneau, S. and others},
	title = {{Overview of the DESI Legacy Imaging Surveys}},
	journal = {Astronomical Journal},
	volume = {157},
	number = {5},
	pages = {168},
	year = {2019},
	issn = {1538-3881},
	publisher = {The American Astronomical Society},
	doi = {10.3847/1538-3881/ab089d}
}

@article{ref_walmsley2023,
	author = {Walmsley, M. and G{\ifmmode\acute{e}\else\'{e}\fi}ron, T. and Kruk, S. and Scaife, A. M. M. and Lintott, C. and Masters, K. L. and Dawson, J. M. and Dickinson, H. and Fortson, L. and Garland, I. L. and others},
	title = {{Galaxy Zoo DESI: Detailed morphology measurements for 8.7M galaxies in the DESI Legacy Imaging Surveys}},
	journal = {Monthly Notices of the Royal Astronomical Society},
	volume = {526},
	number = {3},
	pages = {4768--4786},
	year = {2023},
	issn = {0035-8711},
	publisher = {Oxford Academic},
	doi = {10.1093/mnras/stad2919}
}

@article{ref_smith2024,
	author = {Smith, M. J. and Roberts, R. J. and Angeloudi, E. and Huertas-Company, M.},
	title = {{AstroPT: Scaling Large Observation Models for Astronomy}},
	journal = {ArXiv e-prints},
	year = {2024},
	eprint = {2405.14930},
	doi = {10.48550/arXiv.2405.14930}
}