KokosDev/single-cell-brain-zarr

Single-Cell Brain Zarr Collection

Production-ready brain single-cell RNA-seq data exported from the CellxGene Census into native Zarr stores for chunked, on-demand access on the Hugging Face Hub.

Why Zarr

Single-cell expression matrices get impractical fast if you treat them like ordinary dense files. Zarr is the point of this repo: it makes large atlas-scale data usable without forcing users to download or materialize the whole matrix before they can do anything useful.

• —Compression vs dense float32:
• —Quickstart sample: about 703x smaller.
• —Full collection: about 187x smaller.
• —Sample benchmark recorded during build:
• —Open Zarr group: 0.0014 s
• —Read one 1000 x 1000 chunk: 0.0175 s
• —Practical speed difference:
• —Before Zarr: users are pushed toward full-file or full-matrix workflows.
• —After Zarr: users can open the store, inspect metadata, and read only the chunks they need.

What Is In This Repo

• —brain.zarr
• —Quickstart sample with 150,000 cells.
• —Good for tutorials, schema inspection, and lightweight tests.
• —brain_00000.zarr ... brain_00028.zarr
• —Full production collection.
• —29 row-sharded stores.
• —Most shards contain 1,000,000 cells; the final shard contains 967,109.
• —dataset_summary.json
• —Summary statistics for the sample export.

Source And Provenance

• —Upstream source: CellxGene Census API
• —Census version: 2025-11-08
• —Organism: Homo sapiens
• —Filter used for export: tissue_general == 'brain' and is_primary_data == True
• —Source label in store metadata: cellxgene-census
• —Random seed recorded in store metadata: 42

This repo is a Zarr packaging of the upstream Census data to make browser-friendly, programmatic, chunked access practical on the Hub.

Data Layout

Expression matrix

• —Key: X
• —Dtype: float32
• —Compression: Blosc zstd with bitshuffle
• —Sample chunks: (1000, 1000)
• —Full-store chunks: (256, 61497)

Observation metadata in full stores

• —obs/_index
• —obs/assay
• —obs/cell_type
• —obs/dataset_id
• —obs/disease
• —obs/donor_id
• —obs/n_counts
• —obs/n_genes
• —obs/pct_mito
• —obs/sex
• —obs/tissue

Variable metadata in full stores

• —var/_index
• —var/feature_id
• —var/feature_name
• —var/feature_type

Recommended Usage

• —Use brain.zarr if you want a fast, self-contained sample for development or demos.
• —Use the brain_000xx.zarr stores for full-scale work.
• —Process the full collection shard by shard unless you explicitly have the memory budget to combine everything.
• —Treat X as lazily loaded. Avoid converting the full dataset to one in-memory dense array.

Quick Start

Open the sample store directly from Hugging Face

import fsspec
import zarr

mapper = fsspec.get_mapper(
    "hf://datasets/KokosDev/single-cell-brain-zarr@main/brain.zarr"
)
root = zarr.open_group(mapper, mode="r")

print(root["X"].shape)
print(root["obs/_index"][:5])

Open one full shard

import fsspec
import zarr

shard = "brain_00000.zarr"
mapper = fsspec.get_mapper(
    f"hf://datasets/KokosDev/single-cell-brain-zarr@main/{shard}"
)
root = zarr.open_group(mapper, mode="r")

print(shard, root["X"].shape)
print(root["obs/cell_type"][:5])
print(root["obs/n_counts"][:5])

Iterate over all full shards

import fsspec
import zarr

for i in range(29):
    shard = f"brain_{i:05d}.zarr"
    mapper = fsspec.get_mapper(
        f"hf://datasets/KokosDev/single-cell-brain-zarr@main/{shard}"
    )
    root = zarr.open_group(mapper, mode="r")
    print(shard, root["X"].shape)

Scanpy / AnnData Notes

• —brain.zarr is the safer starting point if you want to materialize an AnnData object locally.
• —The full 29-shard collection is intended for shard-wise workflows, streaming, preprocessing, and atlas-scale analysis.
• —QC-style columns are already included in full stores:
• —n_counts
• —n_genes
• —pct_mito

Intended Use

• —Single-cell analysis and preprocessing
• —Training and evaluation pipelines for biology ML workloads
• —Large-scale feature extraction or embedding jobs
• —Benchmarking chunked I/O and Hub-based data access
• —Scanpy / AnnData workflows that need a small sample plus a scalable full dataset path

Important Notes

• —This repo contains native Zarr stores, not Parquet or CSV exports.
• —The quickstart sample and the full sharded collection serve different purposes and are both intentionally included.
• —If you are benchmarking or building loaders, prefer the sharded stores for realistic large-scale access patterns.

Acknowledgements

Built from the CellxGene Census. Please cite and follow the upstream Census terms, licensing, and attribution requirements when using this data in research or products.