ai-department-lpnu/gtex-single-cell-rnaseq

GTEx Single-Cell RNA-seq Dataset

 

This repository provides tools to create a Hugging Face dataset from GTEx single-nucleus RNA-seq data, transforming the hierarchical H5AD format into a flat, ML-ready structure.

Table of Contents

• —Overview
• —Dataset Schema
• —Data Flow
• —Installation
• —Quick Start
• —Detailed Documentation
• —Data Structure
• —Usage Examples
• —Performance
• —Citation

Overview

Data Source

The data comes from GTEx's snRNA-seq atlas:

• —Source: GTEx Portal
• —Publication: Eraslan et al., Science 2022 - "Single-nucleus cross-tissue molecular reference maps toward understanding disease gene function"
• —Content: 209,126 cells from 16 individuals across 10 tissue types
• —Original Format: H5AD (AnnData)
• —File Size: 1.9 GB (compressed)

Dataset Statistics

Gene Name Mapping

Since expression_data is a flat list of 17,695 values, gene names are provided separately in gene_names.txt (one gene name per line, where line number = gene index).

Example - Finding expression for a specific gene:

from datasets import load_dataset
from huggingface_hub import hf_hub_download

REPO_ID = "ai-department-lpnu/gtex-single-cell-rnaseq"

dataset = load_dataset(REPO_ID, split="train")

gene_names_path = hf_hub_download(
    repo_id=REPO_ID,
    repo_type="dataset",
    filename="data/gene_names.txt",
)
with open(gene_names_path) as f:
    gene_names = [line.strip() for line in f]

gene_to_index = {g: i for i, g in enumerate(gene_names)}

cell = dataset[0]

gene_idx = gene_to_index["SAMD11"]
samd11_expression = cell["expression_data"][gene_idx]

print(f"SAMD11 expression: {samd11_expression}")

Dataset Schema

The dataset includes expression data plus all 48 metadata fields from the original GTEx data:

Core Fields

expression_data

• —Type: List of floats
• —Length: 17,695 values per cell
• —Format: Flat 1D list (no nested structure)
• —Content: Normalized gene expression values
• —Example: [1.23, 0.0, 2.45, ...]

Metadata Fields (48 fields total)

The dataset preserves all metadata from the original H5AD file, including:

Key Biological Metadata

• —tissue: Tissue type (e.g., "Lung", "Heart", "Brain")
• —Age_bin: Donor age range (e.g., "60-69")
• —Sex: Donor biological sex
• —Broad cell type: High-level cell type annotation
• —Granular cell type: Detailed cell type annotation
• —individual: Donor identifier

Sample Information

• —Sample ID: Unique sample identifier
• —Participant ID: GTEx participant identifier
• —Tissue Site Detail: Detailed tissue location
• —prep: Sample preparation batch

Quality Metrics

• —nGenes: Number of genes detected
• —nUMIs: Number of UMIs (transcripts) detected
• —PercentMito: Mitochondrial gene percentage
• —PercentRibo: Ribosomal gene percentage
• —RIN score: RNA integrity number
• —Autolysis Score: Tissue degradation metric
• —Sample Ischemic Time (mins): Time before preservation

Technical Annotations

• —scrublet: Doublet detection flag
• —scrublet_score: Doublet score
• —batch: Sequencing batch
• —barcode: Cell barcode
• —leiden: Leiden clustering assignment
• —leiden_tissue: Tissue-specific clustering

Alignment Statistics

• —introns, junctions, exons: Read mapping stats
• —sense, antisense, intergenic: Read categories
• —exon_ratio, intron_ratio, junction_ratio: Mapping ratios

And more... (see full list of 48 fields in the dataset)

Data Flow

Pipeline Overview

The dataset creation follows a simple 4-step pipeline:

GTEx Portal → Download → Load & Transform → HF Dataset

Key Transformation: Hierarchical HDF5 (209K cells × 17K genes + 48 metadata fields) → Flat structure (49 columns: expression_data + all 48 metadata fields)

Data Transformation Steps

Step 1: H5AD File Structure

GTEx_8_tissues_snRNAseq_atlas_071421.public_obs.h5ad
│
├── X (Expression Matrix)
│   ├── Shape: (209126, 17695)
│   ├── Type: Sparse CSR or Dense Array
│   └── Data: Normalized gene expression
│
├── obs (Cell Metadata) - 209,126 rows × 48 columns
│   ├── tissue          ← All fields preserved
│   ├── Age_bin         ← All fields preserved
│   ├── Sex             ← All fields preserved
│   ├── Broad cell type ← All fields preserved
│   └── ... (44 more fields)
│
└── var (Gene Metadata) - 17,695 rows
    └── Gene names and annotations

Step 2: Load and Extract

# Load H5AD
adata = anndata.read_h5ad(file)

# Convert sparse to dense
expression_matrix = adata.X.toarray()  # (209126, 17695)

# Extract metadata
tissues = adata.obs['tissue'].values   # (209126,)
ages = adata.obs['Age_bin'].values     # (209126,)

Step 3: Transform to Flat Structure

Each cell is transformed from the AnnData structure to a simple dictionary:

# Before: Cell in AnnData
expression_matrix[i]  # numpy array (17695 genes)
metadata[i]           # All 48 metadata fields from obs DataFrame

# After: Flat dictionary
{
    'expression_data': [1.23, 0.0, 2.45, ...],  # List of 17,695 floats
    'tissue': 'Lung',                            # String
    'Age_bin': '60-69',                          # String
    'Sex': 'Female',                             # String
    'Broad cell type': 'Epithelial',             # String
    ... # + 44 more metadata fields
}

Step 4: Create HF Dataset

Dataset({
    features: {
        'expression_data': Sequence(Value('float64')),
        'tissue': Value('string'),
        'Age_bin': Value('string'),
        'Sex': Value('string'),
        'Broad cell type': Value('string'),
        ... # + 44 more metadata fields
    },
    num_rows: 209126
})

Expression Data Format

Each cell's expression_data is a flat list of 17,695 float values:

expression_data = [
    1.234,   # Gene 1 expression
    0.0,     # Gene 2 expression (not expressed)
    2.456,   # Gene 3 expression
    ...      # 17,692 more values
]

Characteristics:

• —Sparsity: ~70-90% zeros (typical for single-cell data)
• —Range: Usually 0 to ~15 (log-normalized values)
• —Type: Float (64-bit in HF dataset)
• —Size: 17,695 values per cell × 8 bytes = ~141 KB per cell

Citation

If you use this dataset, please cite the original GTEx publication:

@article{eraslan2022single,
  title={Single-nucleus cross-tissue molecular reference maps toward understanding disease gene function},
  author={Eraslan, G{\"o}kcen and Drokhlyansky, Eugene and Anand, Shankara and others},
  journal={Science},
  volume={376},
  number={6594},
  pages={eabl4290},
  year={2022},
  publisher={American Association for the Advancement of Science}
}

And the GTEx Consortium:

@article{gtex2020gtex,
  title={The GTEx Consortium atlas of genetic regulatory effects across human tissues},
  author={GTEx Consortium},
  journal={Science},
  volume={369},
  number={6509},
  pages={1318--1330},
  year={2020},
  publisher={American Association for the Advancement of Science}
}

License

The GTEx data is available under the GTEx Data Use Agreement.