NCHC-bio/cell_x_gene_exploratory_data_analysis

feat(data): add new .h5ad sample file for scRNA-seq dataset

fix(gitattributes): add .h5ad file type to LFS tracking

Merge branch 'main' of https://huggingface.co/NCHC-bio/cell_x_gene_visualization into main

feat(visualization): add detail gene visualization reuslt

Upload d7476ae2-e320-4703-8304-da5c42627e71__HTAPP-330-SMP-1082_scRNA-seq.h5ad

fix(notebook): fix the notebook code

feat(eda): normalize dataset paths and deduplicate results in summary

feat(eda): retrieve chunk size for each dataset in batch processing

feat(config): update dataset size thresholds for improved processing efficiency

feat(eda): update large file processing to support parallel workers and enhance metadata caching

feat(config): fix some hard code config and docs

feat(eda): categorize datasets into small, dask-ready, and xlarge for improved processing

feat(eda): adjust worker settings and add emergency mode for handling failed slices for extremly large

feat(eda): enhance dataset processing for extra large

fix(eda): correct max_workers and min_workers values for optimal resource allocation

feat(eda): add adaptive scaling parameters and initial worker configuration for improved resource management

feat(eda): enhance resource utilization by optimizing worker allocation and processing parameters

feat(eda): optimize resource allocation and processing parameters for enhanced performance

feat(metadata): add handling for missing datasets in CELLxGENE metadata and update status reporting

feat(eda): add cache validation and retry mechanism for metadata build

fix(eda): optimize gene statistics calculation in distributed EDA

feat(eda): update resource specifications for optimized performance

feat(slurm): create logs directory and add to gitignore

fix(eda): remove undefined 'info' variable reference causing crash

fix(slurm): correct job time allocation in SLURM script

refactor(slurm): update resource allocation and remove deprecated script

feat(slurm): add SKIP_CACHE_BUILD option to skip metadata cache building

feat(eda): add resume capability and graceful error handling

feat(eda): implement hybrid processing strategy for small and large datasets

feat(eda): refactor distributed EDA script for improved performance and memory management

fix(eda): optimize memory usage and ensure complete data computation

feat(eda): migrate to Dask distributed with adaptive scaling and memory limits

fix(eda): use recent throughput instead of cumulative average for adaptive scaling

feat(eda): add adaptive worker reduction based on throughput monitoring

fix(config): remove mem_per_worker_gib from config files and calculate dynamically in resource_probe script

fix(config): clarify max_memory_gib allocation for staged processing

fix(config): increase max_entries to 1T to include 520B entry dataset

fix(retry): add size categorization after merge to prevent null categories

fix(retry): include corrupted status in retry logic

fix(cache): categorize ok_retry and ok_h5py datasets by size

fix(eda): include all successfully scanned datasets (ok_retry, ok_h5py)

docs(cache): clarify incremental cache behavior and metadata skip option

feat(cache): add enhanced metadata cache to repository

feat(recovery): add corrupted file redownload script and documentation

feat(retry_failed_cache): implement dataset retry mechanism and merging of results

fix(cache): implement two-phase scanning to handle large files serially and prevent OOM

fix(cache): use ProcessPoolExecutor for HDF5 thread-safety

feat(pipeline): add YAML config, metadata-aware scheduling, and dataset slicing

fix(eda): prevent BrokenProcessPool cascade failures

Initial commit: distributed EDA pipeline, max non-zero reporting, and notebook