AstroCompress/SBI-16-2D

SBI-16-2D Dataset

SBI-16-2D is a dataset which is part of the AstroCompress project. It contains imaging data assembled from the Hubble Space Telescope (HST). Note that the underlying data is released under CC-4.0.

Usage

You first need to install the datasets and astropy packages:

pip install datasets astropy

There are two datasets: tiny and full, each with train and test splits. The tiny dataset has 2 2D images in the train and 1 in the test. The full dataset contains all the images in the data/ directory.

Local Use (RECOMMENDED)

You can clone this repo and use directly without connecting to hf:

git clone https://huggingface.co/datasets/AstroCompress/SBI-16-2D

To pull all data files:

git lfs pull

Then cd SBI-16-3D and start python like:

from datasets import load_dataset
dataset = load_dataset("./SBI-16-2D.py", "tiny", data_dir="./data/", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")

Now you should be able to use the ds variable like:

ds["test"][0]["image"].shape # -> (TBD)

Note of course that it will take a long time to download and convert the images in the local cache for the full dataset. Afterward, the usage should be quick as the files are memory-mapped from disk.

Use from Huggingface Directly

This method may only be an option when trying to access the "tiny" version of the dataset.

To directly use from this data from Huggingface, you'll want to log in on the command line before starting python:

huggingface-cli login

or

import huggingface_hub
huggingface_hub.login(token=token)

Then in your python script:

from datasets import load_dataset
dataset = load_dataset("AstroCompress/SBI-16-2D", "tiny", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")

Demo Colab Notebook

We provide a demo collab notebook to get started on using the dataset here.

Utils scripts

Note that utils scripts such as eval_baselines.py must be run from the parent directory of utils, i.e. python utils/eval_baselines.py.