Datasetpublic

LiteFold/Evolutionary

Evolutionary MSA Data This repository contains precomputed evolutionary sequence-alignment data in an archive format that is practical to host and download from the Hub. The original file paths are preserved inside the tar shard, while metadata.csv gives a searchable index of every file. The dataset is meant for workflows that need ready-to-use MSA/cache files without rebuilding them from sequence databases. Contents Component Files Size msa_cache/ 134… See the full description on the dataset page: https://huggingface.co/datasets/LiteFold/Evolutionary.

sourceHugging Faceupdated 4mo agoView on Hugging Face

1likes24downloads

Dataset Card

Evolutionary MSA Data

This repository contains precomputed evolutionary sequence-alignment data in an archive format that is practical to host and download from the Hub. The original file paths are preserved inside the tar shard, while metadata.csv gives a searchable index of every file.

The dataset is meant for workflows that need ready-to-use MSA/cache files without rebuilding them from sequence databases.

Component	Files	Size
`msa_cache/`	134,898	19.37 GiB
`mmseqs30/`	4	82.32 MiB
root files	4	41.99 MiB

File types:

Type	Files	Size
`.npz`	134,898	19.37 GiB
`.fasta`	3	80.71 MiB
`.parquet`	1	34.33 MiB
`.json`	1	7.65 MiB
`.tsv`	1	1.61 MiB
`.sh`	1	806 B
`.md`	1	543 B

Packaging:

Item	Value
Indexed files	134,906
Original payload	19.49 GiB
Tar shards	1
Split large-file parts	0
Archive size	19.72 GiB
Metadata generated	2026-05-25T06:39:53Z

Layout

text

README.md
_MANIFEST.json
metadata.csv
shards.csv
parts.csv
shards/
  shard-00000.tar

Most users only need metadata.csv and shards/shard-00000.tar. The metadata table is configured as the default Dataset Viewer table.

Metadata

metadata.csv has one row per original file.

Column	Meaning
`path`	Original relative path.
`storage_type`	`tar` for files stored inside a tar shard; `parts` for oversized files split into byte parts.
`shard_path`	Tar shard containing the file.
`member_path`	Path of the file inside the tar shard.
`parts_count`	Number of split parts, if any.
`part_paths`	Semicolon-separated part paths, if any.
`top_level`, `directory`, `filename`, `extension`	Path fields for filtering.
`size_bytes`, `size_human`, `modified_utc`	Size and timestamp captured during packaging.

shards.csv lists archive shards. parts.csv is present for consistency with the archive format; this upload does not currently require split parts.

Python API Examples

Install recent Hugging Face clients in your environment:

python

# pip install -U huggingface_hub datasets

Set the repo id once:

python

repo_id = "LiteFold/Evolutionary"

Browse Files

python

from datasets import load_dataset

files = load_dataset(repo_id, "files", split="train")
print(files)
print(files[0])

For streaming access to the index:

python

from datasets import load_dataset

files = load_dataset(repo_id, "files", split="train", streaming=True)
for row in files:
    if row["extension"] == ".npz":
        print(row["path"], row["size_human"], row["shard_path"])
        break

Extract One MSA Cache File

This downloads the shard containing the selected file, then extracts only that member.

python

from pathlib import Path
import tarfile

from datasets import load_dataset
from huggingface_hub import hf_hub_download

repo_id = "LiteFold/Evolutionary"
out_dir = Path("./evolutionary")

files = load_dataset(repo_id, "files", split="train", streaming=True)
row = next(item for item in files if item["extension"] == ".npz")

shard = hf_hub_download(
    repo_id=repo_id,
    repo_type="dataset",
    filename=row["shard_path"],
)

with tarfile.open(shard) as archive:
    archive.extract(row["member_path"], path=out_dir)

print(out_dir / row["path"])

Restore The Full Tree

This downloads the repository snapshot and extracts all tar shards into a local directory.

python

from pathlib import Path
import csv
import tarfile

from huggingface_hub import snapshot_download

repo_id = "LiteFold/Evolutionary"
snapshot = Path(snapshot_download(repo_id=repo_id, repo_type="dataset"))
out_dir = Path("./evolutionary")
out_dir.mkdir(parents=True, exist_ok=True)

for shard in sorted((snapshot / "shards").glob("*.tar")):
    with tarfile.open(shard) as archive:
        archive.extractall(out_dir)

with (snapshot / "metadata.csv").open(newline="") as handle:
    for row in csv.DictReader(handle):
        if row["storage_type"] != "parts":
            continue
        target = out_dir / row["path"]
        target.parent.mkdir(parents=True, exist_ok=True)
        with target.open("wb") as dst:
            for part_path in row["part_paths"].split(";"):
                with (snapshot / part_path).open("rb") as src:
                    while chunk := src.read(8 * 1024 * 1024):
                        dst.write(chunk)

print(out_dir)

Notes

—The archive is uncompressed so individual files can be extracted directly from the tar shard.
—The file paths are kept as they appeared in the source data directory.
—Use metadata.csv as the source of truth for locating files inside shards.