Lexxurius/LMDB_Overhead_Geopose

📦 Geocentric Height Regression Dataset

Large-scale, preprocessed dataset for dense height regression of anthropogenic objects from single satellite RGB imagery. Optimized for streaming PyTorch training with PyArrow and IterableDataset.

📖 Overview

This dataset contains 256×256 patches extracted from satellite RGB imagery and corresponding Above-Ground Level (AGL) height maps (DSM). Each patch includes a binary validity mask and capture metadata, enabling robust training for dense regression tasks in geospatial computer vision.

Key Features:

• —✅ Streaming-Optimized Format: Stored in Apache Parquet for batch-wise reading without loading the entire dataset into RAM.
• —🔍 No-Data Filtering: Patches with >30% invalid/no-data pixels are automatically filtered during preprocessing.
• —🌍 Capture Metadata: Ground Sampling Distance (GSD), off-nadir angle, and scale coefficients are preserved per patch.
• —📈 Precomputed Statistics: Regional and globally pooled mean/std values for precise RGB and height normalization.

📁 Directory Structure

dataset/
                # Preprocessed Parquet files (one per region)
├── dataset_region1.parquet
├── dataset_region2.parquet
└── ...              
└── README.md                   

📊 Data Schema (Parquet)

💡 Decoding Binary Columns: ``python rgb = np.frombuffer(row["image"], dtype=np.uint8).reshape(256, 256, 3) h = np.frombuffer(row["height"], dtype=np.uint16).reshape(256, 256) mask = np.frombuffer(row["mask"], dtype=np.uint8).reshape(256, 256) ``

🛠 Preprocessing Pipeline

1. 1.Source Ingestion: Reads JPEG2000/JP2 (RGB), GeoTIFF (AGL heights), and JSON (metadata).
2. 2.Patch Extraction: Splits images into 256×256 non-overlapping tiles (stride = size).
3. 3.No-Data Filtering: Discards patches where the fraction of NO_DATA_VALUE pixels exceeds NO_DATA_THRESHOLD (default: 0.3).
4. 4.Serialization: Converts NumPy arrays to raw bytes via .tobytes() for compact storage.
5. 5.Streaming Parquet Writes: Incrementally writes batches using pyarrow.parquet.ParquetWriter with Snappy compression.
6. 6.Statistics Aggregation: Computes per-region mean/std, then pools them globally using variance aggregation.

📏 Normalization & Statistics

Global normalization is computed via variance pooling to avoid bias from regional imbalances:

Var_global = Σ( (Var_i + Mean_i²) * N_i ) / N_total  -  Mean_global²

Regional statistics are stored in stats/{region}.json. The training pipeline automatically computes the weighted global mean/std at runtime.

🚀 Quick Start

1. Basic Loading & Decoding (PyArrow + Pandas)

import pyarrow.parquet as pq
import numpy as np

table = pq.read_table("dataset/processed/dataset_region1.parquet")
df = table.to_pandas()

# Decode the first patch
row = df.iloc[0]
rgb  = np.frombuffer(row["image"],  dtype=np.uint8).reshape(256, 256, 3)
h    = np.frombuffer(row["height"], dtype=np.uint16).reshape(256, 256)
mask = np.frombuffer(row["mask"],   dtype=np.uint8).reshape(256, 256)

2. Streaming PyTorch Dataset

Use the provided StreamHeightDataset (see training notebook):

from torch.utils.data import DataLoader

dataset = StreamHeightDataset(
    parquet_files=train_files,
    rgb_mean=RGB_MEAN, rgb_std=RGB_STD,
    h_mean=H_MEAN, h_std=H_STD,
    height_dtype=np.uint16,  # ⚠️ Must match preprocessing output
    mode="train"
)
loader = DataLoader(dataset, batch_size=16, num_workers=2, pin_memory=True, drop_last=True)

3. Training Pipeline

The companion notebook train_height_regression.ipynb includes:

• —DeepLabV3Plus + EfficientNetB3 backbone (segmentation_models_pytorch)
• —MaskedSmoothL1Loss with validity mask support
• —Mixed Precision (torch.cuda.amp) + Gradient Accumulation
• —Weights & Biases logging, automatic best-model checkpointing, and prediction visualization

📦 Requirements

pip install pyarrow pandas numpy torch torchvision \
            segmentation-models-pytorch wandb matplotlib \
            scikit-learn tqdm

💡 Best Practices & Troubleshooting

📜 License & Citation

This dataset is released under the [MIT]. Please cite this repository if used in academic research, competitions, or commercial projects.

📬 Support & Contact

• —🐛 Bug Reports & Feature Requests: Open an issue in the repository
• —💬 Discussion & Experiments: Use the Discussions tab or community forums
• —🌐 Demo Notebook: [Link to Kaggle / Colab]

Designed for production-ready geospatial ML pipelines. Dataset version: `v1.0` 🚀