ferpb/spectralwaste-segmentation

SpectralWaste Segmentation

SpectralWaste Segmentation is the RGB-hyperspectral dataset used for the segmentation experiments in:

SpectralWaste Dataset: Multimodal Data for Waste Sorting Automation Sara Casao, Fernando Peña, Alberto Sabater, Rosa Castillón, Darío Suárez, Eduardo Montijano, and Ana C. Murillo IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.

It contains synchronized RGB and hyperspectral images, semantic segmentation masks for the labeled subset, and a larger unlabeled subset. The images are spatially aligned and resized to 256×256 for training.

• —Project page: https://ropertunizar.github.io/publications/spectralwaste/
• —Paper: https://arxiv.org/abs/2403.18033
• —Dataset code: https://github.com/ferpb/spectralwaste-dataset
• —Paper code: https://github.com/ferpb/spectralwaste-segmentation
• —Zenodo: https://doi.org/10.5281/zenodo.10880544

Dataset

The dataset contains:

Each sample contains synchronized:

• —RGB image
• —Hyperspectral image with 224 bands from approximately 900–1700 nm

Labeled samples additionally contain:

• —RGB semantic segmentation mask
• —Segmentation mask transferred to the hyperspectral image

Classes

The segmentation task contains six foreground waste categories plus background. Background pixels are included during training and when computing per-class metrics as class 0, but the background class is excluded when computing the mean IoU (mIoU) reported in the original experiments.

Segmentation masks are stored as PNG images whose pixel values correspond directly to these class IDs.

Data format

The dataset is distributed as sharded WebDataset archives.

A labeled sample contains:

20220929_02_095520.rgb.png
20220929_02_095520.hsi.tiff
20220929_02_095520.label_rgb.png
20220929_02_095520.label_hsi_lt.png
20220929_02_095520.json

An unlabeled sample contains:

20220929_02_095520.rgb.png
20220929_02_095520.hsi.tiff
20220929_02_095520.json

The modalities are stored as:

Sample metadata

Sample identifiers follow:

YYYYMMDD_SS_HHMMSS

For example:

20220929_02_095520

Each sample includes:

{
  "sample_id": "20220929_02_095520",
  "sequence_id": "20220929_02",
  "timestamp": "2022-09-29T09:55:20+02:00"
}

Samples sharing the same sequence_id belong to the same acquisition session.

Loading

Install:

pip install webdataset numpy pillow tifffile

Load a local set of shards:

import webdataset as wds

dataset = wds.WebDataset(
    "data/train-{000000..000015}.tar"
)

sample = next(iter(dataset))
print(sample.keys())

Decode a labeled sample:

import io
import json

import numpy as np
import tifffile
from PIL import Image


def decode_sample(sample):
    return {
        "rgb": np.asarray(
            Image.open(io.BytesIO(sample["rgb.png"])).convert("RGB")
        ),
        "hsi": tifffile.imread(
            io.BytesIO(sample["hsi.tiff"])
        ),
        "label_rgb": np.asarray(
            Image.open(io.BytesIO(sample["label_rgb.png"]))
        ),
        "label_hsi_lt": np.asarray(
            Image.open(io.BytesIO(sample["label_hsi_lt.png"]))
        ),
        "metadata": json.loads(sample["json"]),
    }

For PyTorch:

import torch

sample = decode_sample(sample)

rgb = torch.from_numpy(sample["rgb"].copy()).permute(2, 0, 1) # (3, 256, 256)
hsi = torch.from_numpy(sample["hsi"].copy()) # (224, 256, 256)
label_rgb = torch.from_numpy(sample["label_rgb"].copy()).long() # (256, 256)
label_hsi_lt = torch.from_numpy(sample["label_hsi_lt"].copy()).long() # (256, 256)

Preprocessing

This dataset is generated from the original SpectralWaste acquisitions using the preprocessing pipeline in the dataset repo.

The pipeline:

• —aligns the RGB and HSI modalities
• —resizes both to 256 × 256
• —converts RGB instance annotations to semantic masks
• —transfers RGB annotations to the HSI modality
• —preserves RGB as 8-bit data and HSI as 16-bit data

The HSI masks transferred labels rather than independently annotated hyperspectral ground truth.

Splits

The labeled samples are divided into training, validation, and test sets using the script available in the dataset repo. The remaining samples form the unlabeled split.

Full-resolution data

This repository is intended for segmentation training and benchmarking.

The original resolution data are available separately and are substantially larger. Users requiring the original spatial resolution or sensor geometry should use the full-resolution release.

License

The dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Citation

If you use this dataset, please cite:

@inproceedings{casao2024spectralwaste,
  title = {{SpectralWaste} Dataset: Multimodal Data for Waste Sorting Automation},
  author = {Casao, Sara and Pe{\~n}a, Fernando and Sabater, Alberto and Castill{\'o}n, Rosa and Su{\'a}rez, Dar{\'i}o and Montijano, Eduardo and Murillo, Ana C.},
  year = {2024},
  booktitle = {2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages = {5852-5858},
  doi = {10.1109/IROS58592.2024.10801797}
}