RosettaCommons/MIP
Microbiome Immunity Project: Protein Universe ~200,000 predicted structures for diverse protein sequences from 1,003 representative genomes across the microbial tree of life and annotate them functionally on a per-residue basis. Quickstart Usage Install HuggingFace Datasets package Each subset can be loaded into python using the Huggingface datasets library. First, from the command line install the datasets library $ pip install datasets Optionally… See the full description on the dataset page: https://huggingface.co/datasets/RosettaCommons/MIP.
11.4k
1 2 3import datasets4import pyarrow5 6def test_local_hf_match(dataset_tag):7 print(f"For dataset : '{dataset_tag}' testing if local and remote ids match ...")8 ids_hf = datasets.load_dataset(9 path = "RosettaCommons/MIP",10 name = dataset_tag,11 data_dir = dataset_tag,12 cache_dir = "/scratch/maom_root/maom0/maom",13 keep_in_memory = True).data['train'].select(['id']).to_pandas()14 ids_local = pyarrow.parquet.read_table(15 source = f"intermediate/{dataset_tag}.parquet",16 columns = ["id"]).to_pandas()17 assert ids_local.equals(ids_hf)18 19 20test_local_hf_match("rosetta_high_quality_models")21test_local_hf_match("rosetta_low_quality_models")22test_local_hf_match("dmpfold_high_quality_models")23test_local_hf_match("dmpfold_low_quality_models")24 25test_local_hf_match("rosetta_high_quality_function_predictions")26test_local_hf_match("rosetta_low_quality_function_predictions")27test_local_hf_match("dmpfold_high_quality_function_predictions")28test_local_hf_match("dmpfold_low_quality_function_predictions")29 30 31 32import pandas33dataset_long = pyarrow.parquet.read_table(34 "intermediate/dmpfold_low_quality_function_predictions.parquet").to_pandas()35 36dataset_wide = pandas.pivot(37 dataset_long[["id", "term_id", "Y_hat"]],38 columns = "term_id",39 index = "id",40 values = "Y_hat")41