flair-bio/bfd

Dataset Card for flair-bio/bfd

Dataset Summary

This dataset is a cleaned, deduplication-clustered, and quality-scored version of the Big Fantastic Database (BFD) — a large metagenomic protein sequence database originally assembled from metaclust and used as an MSA source in AlphaFold. It has been reprocessed by the FLAIR modules/data pipeline into a single training-ready Parquet dataset (sharded), with per-sequence redundancy-reduction (MMseqs2 cascaded clustering) and quality scores (RED — Residue Embedding Diversity) added, for use in pretraining protein language models (pLMs).

• —Curated by: FLAIR (Mila / flair-bio)
• —Source data: BFD — `bfd_metaclust_clu_complete_id30_c90_final_seq`
• —Language(s): Not applicable (protein sequences; FASTA header metadata is English free-text)
• —License: CC BY 4.0 for the FLAIR-processed release.

Dataset Details

Source Data & Provenance

BFD is distributed as a single large tarball of pre-clustered sequences:

https://bfd.mmseqs.com/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt.tar.gz

The tarball is downloaded with aria2c (16-way split, single concurrent download given there is only one source file). It was fetched in August 2026, the most recent version available at that time; BFD is distributed as a single unversioned static archive (no dated release identifier is published upstream), so reproducibility relies on the file remaining unchanged at that URL.

Processing Pipeline

Processed by the shared FLAIR data pipeline (`modules/data`):

Download → Preprocess → Cluster → Score → Assemble → Upload

1. 1.Preprocess — BFD's tarball is streamed and only the *_a3m.ffdata member is extracted (the cs219/hhm profile members are skipped, as they are not raw sequence data). Alignment gaps, consensus rows, and database-match rows are stripped from the A3M records. The remaining sequences are linearized into a monolithic FASTA, split into 1,000,000-sequence chunks, and converted to Parquet shards in parallel. Each sequence is assigned a unique, collision-free ID of the form BFD_000000000000 (a dataset prefix followed by a zero-padded 12-digit counter), with the original FASTA header preserved as description.
2. 2.Cluster — cascaded MMseqs2 (easy-linclust) redundancy reduction at identity thresholds [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3] (coverage threshold 0.8), producing one cluster- representative column per threshold.
3. 3.Score — each sequence is scored with the RED (Residue Embedding Diversity) metric using flair-bio/amplify-350m as the embedding model — a measure of how distinct/informative a sequence's contextual residue embeddings are.
4. 4.Assemble — base sequences, cluster assignments, and RED scores are joined on the shared sequence_id into the final sharded Parquet dataset published here.

Dataset Structure

One train split, stored as multiple Parquet shards (train/*.parquet).

Data Statistics

Computed over the full assembled dataset:

Uses

Intended for self-supervised pretraining (e.g., masked language modeling) and representation learning of protein language models. The cluster_rep_at_* columns allow downstream users to subsample at a desired redundancy level (e.g., dedup at 30% identity for maximum diversity, or 90% for near-duplicate-only removal), and red_score can be used to filter or weight sequences by estimated informational diversity.

Bias, Risks, and Limitations

• —BFD is a metagenomic database; sequence provenance, quality, and annotation completeness vary widely and largely lack experimental validation.
• —RED scores are a heuristic computed from a specific pLM's embeddings (flair-bio/amplify-350m) and should not be interpreted as a ground-truth quality or functional label.
• —Clustering provides an approximate indication of protein relatedness. These datasets use MMseqs2 Linclust, and resulting clusters can vary depending on the clustering algorithm and its parameters.
• —Clustering thresholds reduce but do not eliminate redundancy; sequences sharing a cluster representative at coarse thresholds (e.g., 30%) may still be highly similar within.

Citation

If you use this dataset, please cite the original BFD source, MMseqs2, RED, and FLAIR:

@article{steinegger2017mmseqs2,
  title   = {MMseqs2 enables sensitive protein sequence searching for the analysis of massive sequence datasets},
  author  = {Steinegger, Martin and S{\"o}ding, Johannes},
  journal = {Nature Biotechnology},
  volume  = {35},
  pages   = {1026--1028},
  year    = {2017},
  doi     = {10.1038/nbt.3988}
}

@article{lebreton2026plm,
  title   = {pLM representations unlock metagenomic space beyond homology},
  author  = {Le Breton, Lola and Heurtel-Depeiges, David and Millar, Douglas C. and Zetzsche, Lara E. and Vernon, Robert M. and Langmead, Christopher James and Chandar, Sarath and Fournier, Quentin},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.07.28.739874}
}

@misc{bfd,
  title        = {Big Fantastic Database (BFD)},
  howpublished = {\url{https://bfd.mmseqs.com/}},
}

@misc{flair-plm,
  title        = {FLAIR: Protein Language Model Pretraining Data Pipeline},
  author       = {{Applied Machine Learning Research Team (AMLRT) collaborators at Mila and FLAIR-bio research students}},
  howpublished = {\url{https://github.com/flair-bio/amplify/tree/main/modules/data}},
}