LiteFold/UniProtKB

1---2license: cc-by-4.03pretty_name: UniProtKB Processed4size_categories:5  - 100M<n<1B6task_categories:7  - feature-extraction8language:9  - en10tags:11  - biology12  - proteins13  - uniprot14  - uniprotkb15  - swiss-prot16  - trembl17  - protein-sequences18  - bioinformatics19  - train-validation-test-split20  - jsonl21configs:22  - config_name: default23    data_files:24      - split: train25        path:26          - data/train-*.jsonl.gz27      - split: test28        path:29          - data/test-*.jsonl.gz30  - config_name: sprot31    data_files:32      - split: train33        path:34          - tables/source_set=sprot/split=train/*.jsonl.gz35      - split: validation36        path:37          - tables/source_set=sprot/split=validation/*.jsonl.gz38      - split: test39        path:40          - tables/source_set=sprot/split=test/*.jsonl.gz41  - config_name: sprot_varsplic42    data_files:43      - split: train44        path:45          - tables/source_set=sprot_varsplic/split=train/*.jsonl.gz46      - split: validation47        path:48          - tables/source_set=sprot_varsplic/split=validation/*.jsonl.gz49      - split: test50        path:51          - tables/source_set=sprot_varsplic/split=test/*.jsonl.gz52  - config_name: trembl53    data_files:54      - split: train55        path:56          - tables/source_set=trembl/split=train/*.jsonl.gz57      - split: validation58        path:59          - tables/source_set=trembl/split=validation/*.jsonl.gz60      - split: test61        path:62          - tables/source_set=trembl/split=test/*.jsonl.gz63---64 65# UniProtKB Processed66 67The aim of the UniProt Knowledgebase (UniProtKB; https://www.uniprot.org/) is to provide users with a comprehensive, high-quality and freely accessible set of protein sequences annotated with functional information. In this publication, we describe ongoing changes to our production pipeline to limit the sequences available in UniProtKB to high-quality, non-redundant reference proteomes. We continue to manually curate the scientific literature to add the latest functional data and use machine learning techniques. We also encourage community curation to ensure key publications are not missed. We provide an update on the automatic annotation methods used by UniProtKB to predict information for unreviewed entries describing unstudied proteins. Finally, updates to the UniProt website are described, including a new tab linking protein to genomic information. In recognition of its value to the scientific community, the UniProt database has been awarded Global Core Biodata Resource status.68 69## Dataset Summary70 71| Source set | Description | Protein records |72|---|---|---:|73| `sprot` | Swiss-Prot reviewed canonical proteins | 574,627 |74| `sprot_varsplic` | Swiss-Prot alternative isoform sequences | 41,333 |75| `trembl` | TrEMBL unreviewed proteins | 202,556,314 |76| **Total** |  | **203,172,274** |77 78Additional source totals:79 80| Metric | Value |81|---|---:|82| Total residues | 75,747,523,712 |83| Sequence shards | 205 |84| Protein-entry table shards | 615 |85| Default index rows | 830 |86| Sequence shard bytes | 46,504,287,641 |87| Metadata records bytes | 74,373,082,266 |88| Protein-entry table bytes | 18,549,213,567 |89 90## Default Index Splits91 92The default Dataset Viewer index is split deterministically by `sha256(file_id) % 10`: bucket `0` is `test`, and buckets `1` through `9` are `train`.93 94| Split | Rows |95|---|---:|96| `train` | 733 |97| `test` | 97 |98 99## Protein-Entry Splits100 101The full protein-entry tables use deterministic exact-sequence hash splits. Exact duplicate amino-acid sequences are kept in the same split.102 103| Split | Protein records |104|---|---:|105| `train` | 162,548,965 |106| `validation` | 20,308,533 |107| `test` | 20,314,776 |108 109These are exact-sequence splits, not homology-cluster splits. For strict homology-aware model evaluation, create an additional split using UniRef, MMseqs, or another sequence-clustering method.110 111## Loading With `datasets`112 113Load the default file/table index:114 115```python116from datasets import load_dataset117 118index = load_dataset("LiteFold/UniProtKB")119print(index)120print(index["train"][0])121```122 123Load Swiss-Prot reviewed protein entries:124 125```python126from datasets import load_dataset127 128sprot = load_dataset("LiteFold/UniProtKB", "sprot")129train = sprot["train"]130valid = sprot["validation"]131test = sprot["test"]132```133 134Load Swiss-Prot alternative isoform entries:135 136```python137from datasets import load_dataset138 139isoforms = load_dataset("LiteFold/UniProtKB", "sprot_varsplic")140```141 142Stream TrEMBL entries:143 144```python145from datasets import load_dataset146 147rows = load_dataset("LiteFold/UniProtKB", "trembl", split="train", streaming=True)148for row in rows:149    print(row["accession"], row["protein_name"])150    break151```152 153Use the default index to discover table shards:154 155```python156from datasets import load_dataset157 158index = load_dataset("LiteFold/UniProtKB", split="train")159trembl_train_shards = index.filter(160    lambda row: row["role"] == "protein_entry_table_shard"161    and row["source_set"] == "trembl"162    and row["table_split"] == "train"163)164print(trembl_train_shards[0]["path"])165```166 167## Default Columns168 169| Column | Type | Description |170|---|---|---|171| `file_id` | string | Stable file identifier, currently the repository path. |172| `repo_id` | string | Hugging Face dataset repository id. |173| `source_sha` | string | Source repository commit used to build the index. |174| `dataset_id` | string | Source dataset id from `_MANIFEST.json`. |175| `source_set` | string | `sprot`, `sprot_varsplic`, `trembl`, or empty for repository-level files. |176| `source_slug` | string | Source file slug used in the original manifests. |177| `source_file` | string | Original source file path. |178| `path` | string | Path in this Hugging Face repository. |179| `role` | string | File role such as `protein_entry_table_shard`, `sequence_shard`, or `metadata_records`. |180| `table_split` | string | Protein-entry split for table shards. |181| `shard_index` | int64 | Parsed shard index when present, otherwise `-1`. |182| `size_bytes` | int64 | File size in bytes. |183| `compression` | string | Compression format when applicable. |184| `records_in_source` | int64 | Protein records in the source set, otherwise `-1`. |185| `residues_in_source` | int64 | Residues in the source set, otherwise `-1`. |186| `shards_in_source` | int64 | Number of sequence shards in the source set, otherwise `-1`. |187| `records_in_table_split` | int64 | Protein records in that source set and split, otherwise `-1`. |188| `records_total` | int64 | Total protein records across UniProtKB. |189| `residues_total` | int64 | Total residues across UniProtKB. |190| `total_sequence_shards` | int64 | Total sequence shards. |191| `is_sequence_shard` | bool | Whether the row points to a FASTA sequence shard. |192| `is_table_shard` | bool | Whether the row points to a parsed protein-entry table shard. |193| `is_metadata_records` | bool | Whether the row points to metadata records. |194| `download_pattern` | string | Glob or exact path that can be used for file downloads. |195| `access_note` | string | Short note describing how to load the row's data. |196| `split_bucket` | int64 | Deterministic bucket used for the default train/test split. |197 198## Files199 200- `data/*.jsonl.gz`: default file/table index for Dataset Viewer.201- `tables/source_set=*/split=*/*.jsonl.gz`: full parsed protein-entry tables.202- `sequences/*/*.fasta.zst`: compressed source sequence shards.203- `metadata/*.records.jsonl`: source metadata records.204- `_MANIFEST.json`: source sequence manifest.205- `_POSTPROCESS_MANIFEST.json`: table-generation manifest.206- `dataset_summary.json`: summary of the default index build.207- `scripts/prepare_uniprotkb_dataset.py`: script used to generate the default index.208 209## License210 211CC BY 4.0.212 213## Citation214 215```216@article{uniprot2025,217  title     = {{UniProt}: the {Universal Protein Knowledgebase} in 2025},218  author    = {{The UniProt Consortium}},219  journal   = {Nucleic Acids Research},220  volume    = {53},221  number    = {D1},222  pages     = {D609--D617},223  year      = {2025},224  publisher = {Oxford University Press},225  doi       = {10.1093/nar/gkae1010}226}227```