KokosDev/single-cell-lung-zarr
Single-cell lung (CellxGene Census) — Zarr This dataset was exported from the CellxGene Census as a chunked + compressed Zarr store intended for easy streaming access. Source: CellxGene Census API Organism: Homo sapiens Filter: tissue_general == 'lung' and is_primary_data == True Shape: 100,000 cells × 61,497 genes Zarr path: lung.zarr Compression Uncompressed (dense float32): 22.91 GB Compressed Zarr: ~307 MB (322 MB on Hub) Compression ratio: ~76× (Blosc zstd… See the full description on the dataset page: https://huggingface.co/datasets/KokosDev/single-cell-lung-zarr.
1196
Single-cell lung (CellxGene Census) — Zarr
This dataset was exported from the CellxGene Census as a chunked + compressed Zarr store intended for easy streaming access.
- Source: CellxGene Census API
- Organism: Homo sapiens
- Filter:
tissue_general == 'lung' and is_primary_data == True - Shape:
100,000 cells × 61,497 genes - Zarr path:
lung.zarr
Compression
- Uncompressed (dense float32): 22.91 GB
- Compressed Zarr: ~307 MB (322 MB on Hub)
- Compression ratio: ~76× (Blosc zstd on sparse single-cell data)
The high compression ratio is achieved through Blosc zstd compression optimized for sparse expression matrices typical in single-cell RNA-seq data.
Included labels
obs/cell_typeobs/diseaseobs/tissueobs/sexobs/assay
Cell type distribution (top 25)
Disease labels included
Total unique disease labels: 72
Alzheimer disease
B-cell acute lymphoblastic leukemia
Barrett esophagus
COVID-19
Crohn disease
Down syndrome
HIV infectious disease
HIV infectious disease || leishmaniasis
HIV infectious disease || visceral leishmaniasis
Lewy body dementia
Parkinson disease
Wilms tumor
acute myeloid leukemia
acute promyelocytic leukemia
adenocarcinoma
age related macular degeneration 7
anencephaly
arrhythmogenic right ventricular cardiomyopathy
atrial fibrillation || mitral valve insufficiency
basal cell carcinoma
basal laminar drusen
benign prostatic hyperplasia
breast cancer
breast carcinoma
cataract
chromophobe renal cell carcinoma
clear cell renal carcinoma
colon adenocarcinoma
colon sessile serrated adenoma/polyp
colorectal cancer
common variable immunodeficiency
cytomegalovirus infection
dementia
dilated cardiomyopathy
enamel caries
gastric intestinal metaplasia
gastritis
glioblastoma
hyperplastic polyp
hypertrophic cardiomyopathy
influenza
invasive ductal breast carcinoma
invasive lobular breast carcinoma
leukoencephalopathy, diffuse hereditary, with spheroids 1
long COVID-19
luminal A breast carcinoma
luminal B breast carcinoma
macular degeneration
malignant ovarian serous tumor
multiple sclerosis
myocardial infarction
myocarditis
neuroblastoma
neuroendocrine carcinoma
non-compaction cardiomyopathy
normal
obstructive nephropathy
oral cavity squamous cell carcinoma
pilocytic astrocytoma
plasma cell myeloma
prediabetes syndrome
prostatic acinar adenocarcinoma
pulmonary emphysema
pulpitis
respiratory failure
rheumatoid arthritis
temporal lobe epilepsy
triple-negative breast carcinoma
tubular adenoma
tubulovillous adenoma
type 2 diabetes mellitus
ulcerative colitisData layout
- Expression matrix:
X(dense 2D array) - chunks:
(1000, 1000) - dtype:
float32 - compression: Blosc
zstd - Metadata:
obs/_index,obs/<col>(orobs/<col>_codes+obs/<col>_categoriesfor categoricals)var/_index,var/<col>(orvar/<col>_codes+var/<col>_categoriesfor categoricals)
Loading (zarr → AnnData)
import zarr
import numpy as np
import pandas as pd
from anndata import AnnData
root = zarr.open_group("lung.zarr", mode="r")
X = root["X"] # zarr Array (lazy / on-demand)
obs = pd.DataFrame(index=root["obs/_index"][:].astype(str))
for col in ["cell_type", "disease", "tissue", "sex", "assay"]:
codes_key = "obs/" + col + "_codes"
cats_key = "obs/" + col + "_categories"
if codes_key in root and cats_key in root:
obs[col] = pd.Categorical.from_codes(root[codes_key][:], root[cats_key][:].astype(str))
elif "obs/" + col in root:
obs[col] = root["obs/" + col][:].astype(str)
var = pd.DataFrame(index=root["var/_index"][:].astype(str))
for col in ["feature_name", "feature_id", "gene_symbol"]:
key = "var/" + col
if key in root:
var[col] = root[key][:].astype(str)
# Convert to in-memory AnnData (loads full matrix):
adata = AnnData(X=np.asarray(X), obs=obs, var=var)Scanpy workflow example
import scanpy as sc
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, flavor="seurat_v3")
adata = adata[:, adata.var["highly_variable"]].copy()
sc.pp.scale(adata, max_value=10)
sc.tl.pca(adata, svd_solver="arpack")
sc.pp.neighbors(adata, n_neighbors=15, n_pcs=50)
sc.tl.umap(adata)
sc.tl.leiden(adata, resolution=0.5)Streaming benchmark (single 1000×1000 chunk)
Measured locally while building this dataset:
- open Zarr group: 0.0021 s
- read one
X[chunk](1000×1000): 0.0864 s
{
"open_seconds": 0.0021242350339889526,
"read_chunk_seconds": 0.08641284704208374,
"chunk_shape": [
1000,
1000
],
"chunk_origin": [
84212,
38534
],
"shape": [
100000,
61497
]
}Build metadata
{
"census_version": "2025-11-08",
"created_at": "2026-02-03T06:49:50.898169+00:00",
"max_cells": 100000,
"n_hvg": 0,
"obs_value_filter": "tissue_general == 'lung' and is_primary_data == True",
"organism": "Homo sapiens",
"schema_version": "1.0",
"seed": 0,
"source": "cellxgene-census",
"x_chunks": [
1000,
1000
],
"x_compression": {
"clevel": 3,
"cname": "zstd",
"codec": "blosc",
"shuffle": "bitshuffle"
},
"shape": [
100000,
61497
],
"obs_arrays": [
"_index",
"assay",
"cell_type",
"disease",
"sex",
"tissue"
],
"var_arrays": [
"_index",
"feature_id",
"feature_length",
"feature_name",
"feature_type",
"n_measured_obs",
"nnz",
"soma_joinid"
]
}Upload to HuggingFace Hub
This repo is intended for: KokosDev/single-cell-lung-zarr.
If you have a HuggingFace token locally, upload with:
python3.11 -m pip install huggingface_hub
HF_TOKEN=*** python3.11 build_lung_zarr.py --upload --repo-id "KokosDev/single-cell-lung-zarr"