OneScience-Group/State_datasets

<p align="center"> <strong> <span style="font-size: 30px;">State Dataset</span> </strong> </p>

Dataset Description

State_dataset is a collection of datasets used for State single-cell expression modeling and perturbation prediction tasks. It comprises four data categories: Parse, Tahoe, Replogle-Nadig, and SE-167M-Human. The primary data is in AnnData/H5AD format, accompanied by gene embeddings (PyTorch .pt), dataset split configurations (TOML), and upstream license files.

Supported Tasks

This repository corresponds to the experiment configurations in the State directory: ST-HVG-Parse and ST-SE-Parse use Parse data for few-shot/zero-shot splits by cell type or donor; ST-HVG-Tahoe uses Tahoe data for generalization evaluation; Replogle data is used for perturbation validation; and SE-600M/config.yaml describes the organization of large-scale cellxgene/Tahoe training data and gene embeddings.

Data Format and Structure

The following sizes are based on file statistics from the current directory. File sizes may vary between data versions:

H5AD files can be read with scanpy/anndata, while PT files can be read with PyTorch. The data paths in the configuration files are examples for the runtime environment. After migrating the data to a local environment, update the paths in the State configurations to the actual mount paths.

How to Use the Dataset

Download the dataset:

hf download --dataset OneScience-Group/State_datasets --local-dir ./data

After mounting this directory in the runtime environment, update the data path in the corresponding TOML file to the actual path. For example:

[datasets]
parse = "/path/to/State_dataset/State-Parse-Filtered"

Read an H5AD file:

import anndata as ad

adata = ad.read_h5ad("State-Parse-Filtered/parse_concat_full.h5ad", backed="r")
print(adata)

Sharded Archives

Because the complete directory is approximately 401 GiB, it has been split into multiple Zstandard-compressed shards of 90 GiB (binary) each. The shards are consecutive parts of the same compressed stream and cannot be decompressed independently; they must first be concatenated in order:

cat State_dataset.tar.zst.part-* > State_dataset.tar.zst
zstd -d State_dataset.tar.zst -c | tar -xf -

Alternatively, stream the decompression directly without materializing the merged file:

cat State_dataset.tar.zst.part-* | zstd -d -c | tar -xf -

For shard filenames, actual sizes, and SHA256 checksums, refer to State_dataset.tar.zst.sha256, which was generated in the same directory.

Official OneScience Information

Citation and License

• —Parse data source: Parse Biosciences, “Performance of Evercode WT v3 in Human Immune Cells (PBMCs)”; see State-Parse-Filtered/README.md and CC-NC-4.0-License.txt.
• —For Replogle-Nadig, Tahoe, and SE-167M-Human data, comply with the licenses, citation requirements, and usage restrictions of the respective upstream datasets.
• —This README only describes the current directory structure and does not alter the copyright or license terms of any upstream data.