LiteFold/Mgnify

MGnify Protein Catalogues

The MGnify platform (https://www.ebi.ac.uk/metagenomics) facilitates the assembly, analysis and archiving of microbiome-derived nucleic acid sequences. The platform provides access to taxonomic assignments and functional annotations for nearly half a million analyses covering metabarcoding, metatranscriptomic, and metagenomic datasets, which are derived from a wide range of different environments. Over the past 3 years, MGnify has not only grown in terms of the number of datasets contained but also increased the breadth of analyses provided, such as the analysis of long-read sequences. The MGnify protein database now exceeds 2.4 billion non-redundant sequences predicted from metagenomic assemblies. This collection is now organised into a relational database making it possible to understand the genomic context of the protein through navigation back to the source assembly and sample metadata, marking a major improvement. To extend beyond the functional annotations already provided in MGnify, we have applied deep learning-based annotation methods. The technology underlying MGnify's Application Programming Interface (API) and website has been upgraded, and we have enabled the ability to perform downstream analysis of the MGnify data through the introduction of a coupled Jupyter Lab environment. Oxford Academic In the protein structure prediction context, the MGnify protein database is most commonly used as the deep metagenomic component of MSA pipelines (alongside UniRef and BFD) for AlphaFold2 and related models, where its metagenome-derived sequences enrich poorly represented protein families.

Dataset Summary

The table repo bytes include both top-level table JSONL files and split part files where both are present. The logical table bytes count each upstream table once.

Default Splits

The default index split is deterministic by file id:

sha256(file_id) % 10

Bucket 0 is test; buckets 1 through 9 are train.

These are file-index splits, not biological train/test sequence splits. For model training, create sequence-level or cluster-level splits appropriate to your task after loading the relevant MGnify payload.

Loading With datasets

Load the default file/shard index:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify")
print(index)
print(index["train"][0])

Load one split directly:

from datasets import load_dataset

train_index = load_dataset("LiteFold/Mgnify", split="train")

Find sequence shards for one source family:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify", split="train")
mgy_clusters = index.filter(
    lambda row: row["role"] == "sequence_shard"
    and row["source_family"] == "mgy_clusters"
)
print(mgy_clusters[0]["download_pattern"])

Find split table parts:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify", split="train")
parts = index.filter(lambda row: row["role"] == "table_split_part")
print(parts[0]["path"], parts[0]["size_bytes"])

Streaming Raw FASTA Shards

Download one source family with the Hub client:

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="LiteFold/Mgnify",
    repo_type="dataset",
    allow_patterns=[
        "sequences/sequence_mgnify_current_release_mgy_clusters.fa.gz/shard-*.fasta.zst"
    ],
)
print(local_dir)

Stream a shard without downloading the whole source family:

from huggingface_hub import HfFileSystem
import zstandard as zstd

fs = HfFileSystem()
path = (
    "datasets/LiteFold/Mgnify/"
    "sequences/sequence_mgnify_current_release_mgy_clusters.fa.gz/"
    "shard-000001.fasta.zst"
)

dctx = zstd.ZstdDecompressor()
with fs.open(path, "rb") as f, dctx.stream_reader(f) as reader:
    chunk = reader.read(1 << 20)
    print(chunk[:200])

Downloading Raw Table Parts

For split tables, use the download_pattern column or a direct include pattern:

hf download LiteFold/Mgnify --repo-type dataset \
  --include 'tables/sequence_mgnify_current_release_mgy_proteins_pfam.tsv.gz.jsonl.parts/part-*.jsonl' \
  --local-dir ./mgnify

For unsplit tables:

hf download LiteFold/Mgnify --repo-type dataset \
  --include 'tables/sequence_mgnify_current_release_mgy_seq_metadata_2.tsv.gz.jsonl' \
  --local-dir ./mgnify

The raw table files are not registered as datasets configs because they are multi-TB nested JSONL payloads. Keeping the default config as a compact Parquet index prevents accidental full-repo scans and keeps the Dataset Viewer responsive.

Default Columns

Files

• —data/*.parquet: default file/shard index for Dataset Viewer.
• —metadata/source_files.parquet: full index copy.
• —sequences/*/shard-*.fasta.zst: raw compressed MGnify FASTA shards.
• —tables/**/*.jsonl: raw normalized table payloads and split parts.
• —_MANIFEST.json: index build summary.
• —dataset_summary.json: same summary in a Dataset Viewer-adjacent file.
• —scripts/prepare_mgnify_dataset.py: script used to build the default index.

Source

Derived from LiteFold/Mgnify, originally sourced from EMBL-EBI MGnify.

License

CC BY 4.0.

Citation

@article{richardson2023mgnify,
  title     = {{MGnify}: the microbiome sequence data analysis resource in 2023},
  author    = {Richardson, Lorna and Allen, Ben and Baldi, Germana and Beracochea, Martin and Bileschi, Maxwell L. and Burdett, Tony and Burgin, Josephine and Caballero-P{\'e}rez, Juan and Cochrane, Guy and Colwell, Lucy J. and Curtis, Tom and Escobar-Zepeda, Alejandra and Gurbich, Tatiana A. and Kale, Varsha and Korobeynikov, Anton and Raj, Shriya and Rogers, Alexander B. and Sakharova, Ekaterina and Sanchez, Santiago and Wilkinson, Darren J. and Finn, Robert D.},
  journal   = {Nucleic Acids Research},
  volume    = {51},
  number    = {D1},
  pages     = {D753--D759},
  year      = {2023},
  publisher = {Oxford University Press},
  doi       = {10.1093/nar/gkac1080}
}