Datasetpublic

Chucks90/eryon-data-pipelines

ERYON Data Pipelines Official ERYON ingestion and preprocessing repository. Bucket storage: hf://buckets/Chucks90/eryon-datasets This repo: Chucks90/eryon-data-pipelines Storage Architecture eryon-datasets/ (bucket) ├── raw/ ← raw DICOM / WSIs / genomics archives ├── interim/ ← converted PNGs, tiles, embeddings ├── checkpoints/ ├── inference/ └── simulations/ eryon-data-pipelines/ (this repo) ├── scripts/ ← ingestion + preprocessing scripts… See the full description on the dataset page: https://huggingface.co/datasets/Chucks90/eryon-data-pipelines.

sourceHugging Faceupdated 3mo agoView on Hugging Face

1likes18downloads

Dataset Card

ERYON Data Pipelines

Official ERYON ingestion and preprocessing repository.

Bucket storage: hf://buckets/Chucks90/eryon-datasets This repo: Chucks90/eryon-data-pipelines

Storage Architecture

eryon-datasets/ (bucket)
├── raw/          ← raw DICOM / WSIs / genomics archives
├── interim/      ← converted PNGs, tiles, embeddings
├── checkpoints/
├── inference/
└── simulations/

eryon-data-pipelines/ (this repo)
├── scripts/      ← ingestion + preprocessing scripts
├── manifests/    ← per-dataset JSONL manifests + splits
├── processed/    ← processed dataset records
├── configs/      ← ingestion, preprocessing, validation configs
├── reports/      ← validation + leakage audit reports
└── metadata/     ← dataset_registry.json, version_history.json

Ingestion Flow

TCIA → HF Job → Raw DICOM Download → PNG Conversion
     → Manifest Generation → Leakage Audit
     → Dataset Split → Processed Dataset Repo → Training

Scripts

Script	Purpose
`scripts/lidc_download.py`	Download LIDC-IDRI from TCIA, convert DICOM→PNG, write to bucket
`scripts/manifest_builder.py`	Generate JSONL manifest with sha256, splits, labels
`scripts/split_dataset.py`	Patient-level train/val/test split (no leakage)
`scripts/validate_dataset.py`	Corruption scan, checksum verification, split completeness
`scripts/audit_leakage.py`	Patient overlap, duplicate hash, slice leakage detection

Running the LIDC Pipeline

1. Download via HF Job

bash

hf jobs run \
  --flavor cpu-basic \
  --timeout 12h \
  --secrets HF_TOKEN \
  -v hf://buckets/Chucks90/eryon-datasets:/mnt \
  python:3.12 \
  bash -c "pip install tcia_utils pydicom Pillow -q && python scripts/lidc_download.py"

2. Build manifest

bash

python scripts/manifest_builder.py \
  --root /mnt/raw/lidc \
  --dataset-version 1.0.0 \
  --preprocessing-version 1.0.0 \
  --out manifests/lidc/manifest_v1.0.0.jsonl

3. Split

bash

python scripts/split_dataset.py \
  --manifest manifests/lidc/manifest_v1.0.0.jsonl \
  --seed 42 \
  --out manifests/lidc/splits_v1.0.0.json

4. Validate + audit

bash

python scripts/validate_dataset.py \
  --root /mnt/raw/lidc \
  --manifest manifests/lidc/manifest_v1.0.0.jsonl \
  --splits manifests/lidc/splits_v1.0.0.json \
  --out reports/validation/lidc_v1.0.0.json

python scripts/audit_leakage.py \
  --manifest manifests/lidc/manifest_v1.0.0.jsonl \
  --splits manifests/lidc/splits_v1.0.0.json \
  --out reports/leakage/lidc_v1.0.0.json

Rules

—Raw and processed assets must never be mixed
—Never redownload completed batches (.done sentinels)
—Never run preprocessing inside training pipelines
—Splits are patient-level — no patient spans train/val/test
—Every dataset must have a manifest before training
—Leakage audit is mandatory