Datasetpublic

AstroCompress/GBI-16-4D

GBI-16-4D Dataset GBI-16-4D is a dataset which is part of the AstroCompress project. It contains data assembled from the Sloan Digital SkySurvey (SDSS). Each FITS file contains a series of 800x800 pixel uint16 observations of the same portion of the Stripe82 field, taken in 5 bandpass filters (u, g, r, i, z) over time. The filenames give the starting run, field, camcol of the observations, the number of filtered images per timestep, and the number of timesteps. For example:… See the full description on the dataset page: https://huggingface.co/datasets/AstroCompress/GBI-16-4D.

sourceHugging Facemitupdated 10mo agoView on Hugging Face

1likes987downloads

Dataset Card

GBI-16-4D Dataset

GBI-16-4D is a dataset which is part of the AstroCompress project. It contains data assembled from the Sloan Digital SkySurvey (SDSS). Each FITS file contains a series of 800x800 pixel uint16 observations of the same portion of the Stripe82 field, taken in 5 bandpass filters (u, g, r, i, z) over time. The filenames give the starting run, field, camcol of the observations, the number of filtered images per timestep, and the number of timesteps. For example:

cube_center_run4203_camcol6_f44_35-5-800-800.fits```


contains 35 frames of 800x800 pixel images in 5 bandpasses starting with run 4203, field 44, and camcol 6. The images are stored in the FITS standard.

Note that the underlying data is released under [CC-4.0](https://creativecommons.org/licenses/by/4.0/deed.en).

# Usage

You first need to install the `datasets` and `astropy` packages:

pip install datasets astropy


There are two datasets: `tiny` and `full`, each with `train` and `test` splits. The `tiny` dataset has 2 4D images in the `train` and 1 in the `test`. The `full` dataset contains all the images in the `data/` directory.

## Local Use (RECOMMENDED)

You can clone this repo and use directly without connecting to hf:

git clone https://huggingface.co/datasets/AstroCompress/GBI-16-4D

git lfs pull


Then `cd GBI-16-4D` and start python like:

from datasets import loaddataset dataset = loaddataset("./GBI-16-4D.py", "tiny", datadir="./data/", writerbatchsize=1, trustremotecode=True) ds = dataset.withformat("np")


Now you should be able to use the `ds` variable like:

ds["test"][0]["image"].shape # -> (55, 5, 800, 800)


Note of course that it will take a long time to download and convert the images in the local cache for the `full` dataset. Afterward, the usage should be quick as the files are memory-mapped from disk.


## Use from Huggingface Directly

This method may only be an option when trying to access the "tiny" version of the dataset.

To directly use from this data from Huggingface, you'll want to log in on the command line before starting python:

huggingface-cli login

or

import huggingfacehub huggingfacehub.login(token=token)


Then in your python script:

from datasets import loaddataset dataset = loaddataset("AstroCompress/GBI-16-4D", "tiny", writerbatchsize=1, trustremotecode=True) ds = dataset.with_format("np")



## Demo Colab Notebook
We provide a demo collab notebook to get started on using the dataset [here](https://colab.research.google.com/drive/1SuFBPZiYZg9LH4pqypc_v8Sp99lShJqZ?usp=sharing).

## Utils scripts
Note that utils scripts such as `eval_baselines.py` must be run from the parent directory of `utils`, i.e. `python utils/eval_baselines.py`.