MTT69/TurbulentChannel

PIVtools Turbulent Channel Validation Dataset

Reproducibility capsule for the PIVtools software paper (SoftwareX, submitted). It contains everything needed to regenerate the paper's validation figures, at three levels of effort:

1. 1.Plot only — run two scripts against the shipped reference_results/. No PIV processing.
2. 2.Reprocess — run PIV → calibration → statistics from the shipped images and configs, then plot from your own outputs.
3. 3.Re-derive calibration — recompute the camera models from the shipped calibration board images before reprocessing.

Two synthetic cases and one experimental case:

• —Case A (clean, synthetic): 85 000 particles per 2048 × 2048 image (≈ 5.2 ppw at 16 × 16 windows), no noise. Upper bound on PIV accuracy. Planar + stereo geometries.
• —Case B (noisy, synthetic): 22 000 particles per image (≈ 1.3 ppw), Gaussian sensor noise (mean 80, std 16, SNR ≈ 8, independent per frame). Planar + stereo geometries.
• —Experimental: 2 100 pairs of cropped, gain-normalised Cam4 images of a turbulent channel/boundary-layer facility, compared against a hot-wire profile from the same facility (shipped in hotwire/hwa.mat).

Each synthetic case has 4 000 image pairs. The stereo cameras sit symmetrically at ± 45° from the sheet normal in a side-scatter arrangement. Ground truth comes from DNS-derived particle statistics (JHTDB channel flow, Re_τ ≈ 1000); both cases derive from the same snapshots at their respective particle counts, so finite-sample statistics are self-consistent.

Contents

MTT69/TurbulentChannel/
├── README.md                          (this file)
├── LICENSE                            (CC-BY-4.0)
├── configs/                           ready-to-run PIVtools configs (edit the
│   │                                   C:/EDIT_ME/... paths at the top, nothing else)
│   ├── planar_clean.yaml              one config: clean case uses no preprocessing anywhere
│   ├── planar_noisy_instantaneous.yaml   noisy case ships PER-PIPELINE configs: the
│   ├── planar_noisy_ensemble.yaml        instantaneous ran with ssmin, the ensemble on RAW images
│   ├── stereo_clean.yaml
│   ├── stereo_noisy.yaml
│   ├── experimental_instantaneous.yaml   experimental likewise: ssmin for instantaneous,
│   └── experimental_ensemble.yaml        no filters for the ensemble
├── ground_truth/
│   ├── clean/direct_stats.mat         DNS statistics for Case A (85k particles)
│   └── noisy/direct_stats.mat         DNS statistics for Case B (22k particles)
├── hotwire/
│   └── hwa.mat                        hot-wire profile (uncorrected variance) + u_tau, l+, nu
├── calibration/
│   ├── planar/                        ChArUco boards (calib%05d.tif, 20 views)
│   │   └── calibration/               + fitted model and settings sidecar (dt = 0.0057553 s)
│   ├── stereo/                        board view-pairs under Cam1/, Cam2/ + ground_truth.npz
│   │   └── calibration/               + fitted stereo model and sidecar (dt = 0.0057553 s)
│   └── experimental/                  single board view (calib_Cam_B00001_crop.tif)
│       └── calibration/               + scale-factor model (16.72 px/mm, wall origin,
│                                        axis signs) and sidecar (dt = 4.96e-05 s)
├── reference_results/                 final calibrated outputs of the published runs
│   ├── planar_clean/                  mean_stats.mat + ensemble_result.mat + coordinates.mat
│   ├── planar_noisy/                  (same three files)
│   ├── stereo_clean/                  mean_stats.mat (stereo Cam1_Cam2 statistics)
│   ├── stereo_noisy/                  mean_stats.mat
│   └── experimental/                  mean_stats.mat + ensemble_result.mat + coordinates.mat
├── planar_clean/                      Case A planar images: B00001_A.tif ... B04000_B.tif
├── planar_noisy/                      Case B planar images (same naming)
├── stereo_clean/Cam1/, Cam2/          Case A stereo images (4000 pairs per camera)
├── stereo_noisy/Cam1/, Cam2/          Case B stereo images
├── experimental/cam4_normalised/      2100 pairs of cropped, gain-normalised TIFFs
└── scripts/
    ├── paper_figures.py               synthetic validation figures (Cases A + B vs DNS)
    ├── piv_vs_hwa_paper.py            experimental figure (PIV vs hot wire)
    └── benchmark_comparison.py        shared library (GT loading, wall-unit conversion)

Tier 1 — regenerate the paper figures (no PIV processing)

Only needs standard scientific Python — no PIVtools install:

pip install numpy scipy h5py matplotlib
hf download MTT69/TurbulentChannel --repo-type dataset --local-dir tc
cd tc

# synthetic figures (mean velocity, stresses, combined stresses)
python scripts/paper_figures.py -o figs \
    --gt-clean-dir ground_truth/clean --gt-noisy-dir ground_truth/noisy \
    --inst-clean-stats reference_results/planar_clean/mean_stats.mat \
    --ens-clean-dir   reference_results/planar_clean \
    --stereo-clean-stats reference_results/stereo_clean/mean_stats.mat \
    --inst-noisy-stats reference_results/planar_noisy/mean_stats.mat \
    --ens-noisy-dir   reference_results/planar_noisy \
    --stereo-noisy-stats reference_results/stereo_noisy/mean_stats.mat

# experimental figure (PIV vs hot wire)
python scripts/piv_vs_hwa_paper.py \
    --ens-dir reference_results/experimental \
    --inst-stats reference_results/experimental/mean_stats.mat \
    --hwa hotwire/hwa.mat \
    --ens-pass 2 --inst-pass 2 3 \
    --inst-labels '16$\times$96 px' '8$\times$96 px' \
    --utau-piv 0.7 --utau-hwa 0.68138107 --y-lo 0.640 \
    --out figs/piv_vs_hwa_comparison.png

These exact commands, run against the shipped reference_results/, reproduce the paper figures (fonts permitting: the scripts prefer CMU Serif and fall back to DejaVu). Quote the --inst-labels arguments exactly as shown (single quotes) so your shell does not eat the $ characters.

Tier 2 — reprocess from images

Install PIVtools, then run one case per working directory. The CLI reads config.yaml from the current working directory — no flags, no fallbacks.

pip install pivtools
mkdir work_planar_clean && cd work_planar_clean
cp ../tc/configs/planar_clean.yaml config.yaml
# open config.yaml, replace every C:/EDIT_ME/... path (instructions in the header),
# then:

planar_clean (one config, both pipelines):

pivtools-cli instantaneous
pivtools-cli ensemble
pivtools-cli apply-calibration --all-paths --type-name instantaneous
pivtools-cli apply-calibration --all-paths --type-name ensemble
pivtools-cli statistics

planar_noisy and experimental ship TWO configs each, because the published runs used different preprocessing per pipeline (instantaneous: ssmin 7×7; ensemble: raw images — adding a filter corrupts the ensemble stresses). Use one working directory per config, with the SAME base_paths in both so outputs merge:

# directory 1: <case>_instantaneous.yaml as config.yaml
pivtools-cli instantaneous
pivtools-cli apply-calibration --all-paths --type-name instantaneous
pivtools-cli statistics

# directory 2: <case>_ensemble.yaml as config.yaml
pivtools-cli ensemble
pivtools-cli apply-calibration --all-paths --type-name ensemble

Stereo cases (instantaneous only — the paper uses no stereo ensemble):

pivtools-cli instantaneous
pivtools-cli apply-stereo --all-paths
pivtools-cli statistics --source-endpoint stereo

Outputs land under the base_paths folder you set: calibrated vectors in calibrated_piv/<n>/..., statistics in statistics/<n>/.../mean_stats/mean_stats.mat (stereo: statistics/<n>/stereo/Cam1_Cam2/...). Point the Tier-1 figure commands at your own outputs instead of reference_results/ (the --ens-*-dir arguments want the folder holding ensemble_result.mat + coordinates.mat; the --*-stats arguments want a mean_stats.mat).

GUI instead of CLI: load the same config.yaml in the PIVtools GUI and run the same stages from their tabs (Process → Calibration → Statistics) — identical engine, identical outputs. The calibration tab picks up the shipped model through the same calibration_sources path in the config.

Performance benchmark (optional)

The paper's strong-scaling benchmark is packaged with PIVtools as the pivtools-bench console command — no scheduler scripts required. From the same path-edited planar_clean working directory as Tier 2:

pivtools-bench scaling --config config.yaml \
    --dataset <your download of TurbulentChannel/planar_clean> \
    --n-images 3840 --sweep workers            # add --dry-run to preview the matrix

This runs the full production pipeline (Dask cluster, load, correlate, save) at a geometric sweep of worker counts sized to your machine's cores, two OpenMP threads per worker, at fixed total workload — a strong-scaling measurement. Add --iterations 3 for error bars. Three outputs land in bench_results/ under the working directory:

(Two additional diagnostic figures are also written; the matrix/oversub sweep modes explore worker×thread trade-offs beyond the paper's sweep — see pivtools-bench scaling --help.)

Throughput numbers are hardware-dependent: the paper's peak of 117 pairs/s was measured on a 192-core dual-EPYC node, and your core count sets your ceiling.

Tier 3 — re-derive the calibration models

The shipped models under calibration/*/calibration/ reproduce the paper as-is. To rebuild them from the board images instead:

# planar (single camera; ChArUco 10 x 7 squares, 12 mm, DICT_4X4_1000)
pivtools-cli detect-charuco --source ./tc/calibration/planar \
    --camera 1 --image-format "calib%05d.tif" --n-views 20 --square-size 0.012

# stereo pair (boards under Cam1/, Cam2/)
pivtools-cli detect-stereo --source ./tc/calibration/stereo --camera-pair 1,2 \
    --image-format "calib%05d.tif" --n-views 20 --square-size 0.012

The synthetic boards are rendered through the same camera models that rendered the particle images, so the recovered calibration corresponds exactly to the imaging system (calibration/stereo/ground_truth.npz) holds the true cameras.

The experimental case uses a uniform scale-factor model rather than a fitted camera. The shipped model (calibration/experimental/calibration/Cam1/scale_factor_planar/model/) carries everything: 16.72 px/mm, the wall origin (originpx [254.41, 1626.65] → originmm [0, 11.17]) and the axis directions (colsign −1, rowsign −1). Rebuilding it requires clicking the origin on the board image — use the GUI Scale Factor tab, or pivtools-cli scale-factor --help with the constants above.

Key constants

Image specifications (synthetic)

Ground truth

One file per case, each in its own subdirectory so the figure script can point at them via --gt-clean-dir / --gt-noisy-dir:

• —ground_truth/clean/direct_stats.mat — Case A reference (85 000 particle trajectories)
• —ground_truth/noisy/direct_stats.mat — Case B reference (22 000 trajectories)

Both are computed directly from the JHTDB particle position snapshots used to render the corresponding images (the truth derives from the particle files, not the rendered images). The stereo cases share the same files. Schema:

Hot-wire data

hotwire/hwa.mat is an HDF5 file with a caseData group holding exactly what the experimental figure needs:

Provenance

The synthetic images were rendered from JHTDB turbulent-channel particle trajectories with the EUROPIV Synthetic Image Generator; noise was added post-render with independent per-frame seeds. The full generation chain (JHTDB download, particle conversion, 85k → 22k downsampling, SIG configs, render recipes, noise and ground-truth scripts) was removed from the current tree to keep the capsule focused, but remains available in this repository's git history.

Citation

If you use this dataset, please cite the PIVtools paper.

@article{taylor_pivtools,
  title={PIVtools: A comprehensive open source software for particle image velocimetry},
  author={Taylor, M.T. and Lawson, J.M. and Ganapathisubramani, B.},
  journal={SoftwareX},
  note={submitted}
}

License

CC-BY-4.0 — free to use, modify, and redistribute with attribution to the PIVtools paper.

The DNS ground truth is derived from publicly accessible Johns Hopkins Turbulence Database data (http://turbulence.pha.jhu.edu) and is redistributed here under the same permissive terms.