Modelpublic

NCHC-bio/cell_x_gene_exploratory_data_analysis

sourceHugging Faceupdated 7mo agoView on Hugging Face

0likes

Model Card

Distributed EDA for Cell x Gene

This folder includes a metadata-aware EDA pipeline for large .h5ad files with YAML-based configuration.

All commands below assume your current directory is:

bash

cd /project/GOV108018/whats2000_work/cell_x_gene_visualization

Quick Start

0) Check system resources (optional)

Probe your system and get recommendations for config settings:

bash

# Using your config file
uv run python scripts/resource_probe.py --config configs/eda_optimized.yaml

# Or specify workdir manually
uv run python scripts/resource_probe.py --workdir /project/GOV108018

This outputs recommendations for workers, memory, and sharding based on your system.

1) Configure pipeline

Use the optimized config (auto-generated for your system: 394 GB RAM, 56 cores):

bash

cat configs/eda_optimized.yaml

Or create your own based on the template:

bash

cp configs/eda_config_template.yaml configs/my_config.yaml
# Edit my_config.yaml with your paths and resource limits

2) Build metadata cache

Pre-scan all datasets to determine sizes and enable intelligent scheduling:

bash

uv run python scripts/build_metadata_cache.py --config configs/eda_optimized.yaml

This creates output/cache/enhanced_metadata.parquet with:

—Dataset dimensions (nobs × nvars)
—File sizes
—Size categories (small/medium/large/xlarge)
—Estimated memory requirements

Cache is incremental - only new/changed files are rescanned. Use --force-rescan to rebuild.

Handling corrupted files: If some files fail during scanning (status='failed' or 'corrupted'), you can:

1.Retry them with a more robust strategy: uv run python scripts/retry_failed_cache.py --cache output/cache/enhanced_metadata.parquet
2.Re-download corrupted files from CELLxGENE: uv run python scripts/redownload_corrupted.py --config configs/eda_optimized.yaml

See Troubleshooting for details.

3) Run EDA pipeline

If cache already built, run EDA directly:

bash

# Run EDA only (skip metadata step)
uv run python scripts/run_eda_pipeline.py --config configs/eda_optimized.yaml --step eda

Or run all steps (cache is incremental, so safe to re-run):

bash

# Full pipeline: metadata + EDA + merge
uv run python scripts/run_eda_pipeline.py --config configs/eda_optimized.yaml

Individual steps:

bash

# Step 1: Build metadata (incremental - only new/changed files)
uv run python scripts/run_eda_pipeline.py --config configs/eda_optimized.yaml --step metadata

# Step 2: Run EDA only
uv run python scripts/run_eda_pipeline.py --config configs/eda_optimized.yaml --step eda

# Step 3: Merge shards (if using sharding)
uv run python scripts/run_eda_pipeline.py --config configs/eda_optimized.yaml --step merge

4) Direct script usage

For more control:

bash

# Build metadata cache
uv run python scripts/build_metadata_cache.py --config configs/eda_optimized.yaml

# Run EDA with all workers
uv run python scripts/distributed_eda.py --config configs/eda_optimized.yaml

# Override worker count
uv run python scripts/distributed_eda.py --config configs/eda_optimized.yaml --force-workers 32

Distributed Processing (SLURM)

For multi-node HPC clusters, use array jobs:

bash

# Submit 4 parallel jobs
sbatch --array=0-3 scripts/run_eda_slurm.sh configs/eda_optimized.yaml 4

# After all jobs complete, merge results
uv run python scripts/merge_eda_shards.py --output-dir output/eda

Or configure sharding in YAML:

yaml

sharding:
  enabled: true
  num_shards: 4
  shard_index: 0  # Override with --shard-index on command line
  strategy: "size_balanced"  # Distribute by size for load balancing

Then run each shard:

bash

uv run python scripts/distributed_eda.py --config configs/eda_optimized.yaml --num-shards 4 --shard-index 0
uv run python scripts/distributed_eda.py --config configs/eda_optimized.yaml --num-shards 4 --shard-index 1
# ... etc

Configuration Guide

Resource Management

The pipeline respects your resource limits and adapts processing strategy by dataset size:

yaml

resources:
  max_memory_gib: 240      # Total memory available
  max_workers: 42          # Maximum parallel workers
  min_workers: 8           # Starting worker count
  chunk_size: 12288        # Matrix chunk size
  
  # Performance tuning
  slowdown_threshold: 0.5  # Reduce workers if throughput drops by 50%
  min_workers_ratio: 0.25  # Keep at least 25% of max workers
  large_file_threshold_gib: 30.0  # Process files >30GB serially

dataset_thresholds:
  small: 2_000_000_000      # < 2B entries
  medium: 15_000_000_000    # < 15B entries  
  large: 40_000_000_000     # < 40B entries (Dask safe)
  max_entries: 1_000_000_000_000  # Reject larger datasets

strategy:
  small:
    chunk_size_multiplier: 1.0     # Use full chunk size
  
  medium:
    chunk_size_multiplier: 1.0     # Full chunk size for efficiency
  
  large:
    chunk_size_multiplier: 0.9     # Slightly reduced for stability
  
  xlarge:
    chunk_size_multiplier: 0.2     # Very small chunks for sequential processing

Key parameters explained:

—slowdown_threshold: If processing throughput drops below this fraction of baseline (e.g., 0.5 = 50%), the pipeline automatically reduces worker count to prevent memory thrashing
—min_workers_ratio: Minimum workers to keep as a fraction of max_workers (e.g., 0.25 = always keep at least 25% of workers)
—large_file_threshold_gib: Files larger than this size (in GB) use a separate worker pool during metadata cache building
—large_file_workers: Number of parallel workers for large files (0 = serial processing, >0 = parallel). With high RAM systems, using 4-8 workers significantly speeds up cache building
—chunk_size_multiplier: Adjusts the chunk size based on dataset category (smaller = safer for large datasets)

Dataset Slicing

Large datasets are automatically sliced to respect memory limits:

yaml

slicing:
  enabled: true
  obs_slice_size: 300000  # Default for medium/large datasets
  obs_slice_size_xlarge: 100000  # Xlarge datasets use smaller slices
  overlap: 0
  merge_strategy: "combine"  # Combine slice statistics

Metadata Integration

Point to CELLxGENE metadata CSVs for enhanced context:

yaml

paths:
  metadata_csvs:
    - /project/GOV108018/cell_x_gene/metadata/dataset_metadata_homo_sapiens.csv
    - /project/GOV108018/cell_x_gene/metadata/dataset_metadata_mus_musculus.csv
  enhanced_metadata_cache: output/cache/enhanced_metadata.parquet

The pipeline merges this with quick-scanned dimensions for intelligent scheduling.

Processing Strategy

The pipeline uses parallel processing with priority ordering:

1.Pre-scan phase: Quick metadata scan (no matrix loading) categorizes datasets by size
2.Parallel execution: All datasets process in parallel using full worker pool
3.Smart ordering: Small datasets (priority 1) start first for quick wins
4.Automatic slicing: Large datasets split into memory-safe chunks
5.Resource-aware: Strategies adapt chunk sizes based on dataset category

This approach fully leverages all available cores throughout the entire pipeline.

Outputs

—Per shard summary CSV:
—output/eda/eda_summary_shard_XXX_of_YYY.csv
—Per shard failure log:
—output/eda/eda_failures_shard_XXX_of_YYY.json
—Per dataset JSON details:
—output/eda/per_dataset/*.json
—Merged summary (after sharding):
—output/eda/eda_summary_all_shards.csv
—Global max report:
—output/eda/max_nonzero_gene_count_all_cells.csv
—output/eda/max_nonzero_gene_count_all_cells.json
—Metadata cache:
—output/cache/enhanced_metadata.parquet

Output Schema

Each dataset result includes:

—Dimensions: nobs, nvars, total_entries
—Sparsity: nnz, sparsity
—Cell statistics: cellsum, cell_nnz_ (mean/std/min/max/quantiles)
—Matrix statistics: xmean, xstd
—Metadata summaries: obs/var column types and top values
—Schema: Complete column names and dtypes
—Processing info: sizecategory, processingmode (full/sliced), elapsed_sec

Visualization

Open the notebook:

bash

uv run jupyter lab notebooks/max_nonzero_gene_report.ipynb

The notebook provides:

—Global max non-zero gene count
—Distribution of cell-level statistics
—Dataset size analysis
—Processing time comparisons

Troubleshooting

Corrupted or failed datasets

If the metadata cache builder reports failed or corrupted files:

Step 1: Retry with robust strategy

Some files may fail due to transient issues or need special handling:

bash

# Using config file (recommended - uses paths and thresholds from config)
uv run python scripts/retry_failed_cache.py --config configs/eda_optimized.yaml

# Or specify paths manually
uv run python scripts/retry_failed_cache.py \
  --cache output/cache/enhanced_metadata.parquet \
  --h5ad-dirs /project/GOV108018/cell_x_gene/homo_sapiens/h5ad \
              /project/GOV108018/cell_x_gene/mus_musculus/h5ad

This script:

—Retries failed datasets with progressively safer strategies (anndata backed mode → h5py direct)
—Categorizes truly corrupted files (truncated/damaged HDF5 structure)
—Merges retry results back into the cache
—Reports final statistics (successful recoveries vs truly corrupted)

Step 2: Re-download corrupted files

For files that are truly corrupted (status='corrupted'), re-download fresh copies from CELLxGENE:

bash

uv run python scripts/redownload_corrupted.py --config configs/eda_optimized.yaml

This script:

—Identifies corrupted files from the metadata cache
—Looks up dataset IDs and download URLs from CELLxGENE metadata CSVs
—Downloads files to output/temp/ for safety
—Verifies each downloaded file is valid HDF5
—Moves verified files to replace corrupted originals
—Keeps failed downloads in temp for inspection

After re-downloading, rebuild the metadata cache to update the status:

bash

uv run python scripts/build_metadata_cache.py --config configs/eda_optimized.yaml --force-rescan

Typical corruption causes:

—Interrupted downloads during dataset collection
—HDF5 file not properly closed/finalized during creation
—Storage/filesystem errors
—Network transfer errors from original source

Note: Files marked as 'corrupted' have HDF5 structural issues (truncated superblock, missing data blocks) and cannot be repaired - they must be re-downloaded from the source.

Metadata cache not found

bash

# Build it first
uv run python scripts/build_metadata_cache.py --config configs/eda_optimized.yaml

Memory errors

Reduce workers and chunk size in config:

yaml

resources:
  max_workers: 24
  chunk_size: 4096
  large_file_threshold_gib: 20.0  # Process larger files separately
  large_file_workers: 2  # Fewer workers for large files (or 0 for serial)
slicing:
  obs_slice_size: 50000

Performance degradation during processing

The pipeline automatically reduces workers if throughput drops significantly. Adjust sensitivity:

yaml

resources:
  slowdown_threshold: 0.3  # More sensitive (30% slowdown triggers reduction)
  min_workers_ratio: 0.5   # Keep more workers (50% minimum)

Dataset too large

Adjust thresholds or enable more aggressive slicing:

yaml

dataset_thresholds:
  max_entries: 50_000_000_000  # Lower limit
slicing:
  obs_slice_size: 30000  # Smaller slices