Lab-Rasool/TCGA

1---2configs:3- config_name: clinical4  data_files:5  - split: gatortron6    path: Clinical Data (gatortron-base)/*7  - split: medgemma8    path: Clinical Data (medgemma)/*9  - split: qwen10    path: Clinical Data (qwen)/*11  - split: llama12    path: Clinical Data (llama)/*13- config_name: pathology_report14  data_files:15  - split: gatortron16    path: Pathology Report (gatortron-base)/*17  - split: medgemma18    path: Pathology Report (medgemma)/*19  - split: qwen20    path: Pathology Report (qwen)/*21  - split: llama22    path: Pathology Report (llama)/*23- config_name: wsi24  data_files:25  - split: uni26    path: Slide Image (UNI)/*27- config_name: molecular28  data_files:29  - split: senmo30    path: Molecular (SeNMo)/*31- config_name: radiology32  data_files:33  - split: remedis34    path: Radiology (REMEDIS)/*35  - split: radimagenet36    path: Radiology (RadImageNet)/*37language:38- en39tags:40- medical41- multimodal42- tcga43- oncology44pretty_name: TCGA45license: cc-by-nc-nd-4.046---47 48# Dataset Card for The Cancer Genome Atlas (TCGA) Multimodal Dataset49<!-- Provide a quick summary of the dataset. -->50 51The Cancer Genome Atlas (TCGA) Multimodal Dataset is a comprehensive collection of clinical data, pathology reports, slide images, molecular data, and radiology images for cancer patients. 52This dataset aims to facilitate research in multimodal machine learning for oncology by providing embeddings generated using state-of-the-art models including GatorTron, MedGemma, Qwen, Llama, UNI, SeNMo, REMEDIS, and RadImageNet.53 54- **Curated by:** Lab Rasool55- **Language(s) (NLP):** English56 57 58## Uses59<!-- Address questions around how the dataset is intended to be used. -->60 61```python62from datasets import load_dataset63 64# Clinical data embeddings (4 models available)65clinical_gatortron = load_dataset("Lab-Rasool/TCGA", "clinical", split="gatortron")66clinical_medgemma = load_dataset("Lab-Rasool/TCGA", "clinical", split="medgemma")67clinical_qwen = load_dataset("Lab-Rasool/TCGA", "clinical", split="qwen")68clinical_llama = load_dataset("Lab-Rasool/TCGA", "clinical", split="llama")69 70# Pathology report embeddings (4 models available)71pathology_gatortron = load_dataset("Lab-Rasool/TCGA", "pathology_report", split="gatortron")72pathology_medgemma = load_dataset("Lab-Rasool/TCGA", "pathology_report", split="medgemma")73pathology_qwen = load_dataset("Lab-Rasool/TCGA", "pathology_report", split="qwen")74pathology_llama = load_dataset("Lab-Rasool/TCGA", "pathology_report", split="llama")75 76# Whole slide image embeddings77wsi_dataset = load_dataset("Lab-Rasool/TCGA", "wsi", split="uni")78 79# Molecular data embeddings80molecular_dataset = load_dataset("Lab-Rasool/TCGA", "molecular", split="senmo")81 82# Radiology embeddings (2 models available)83radiology_remedis = load_dataset("Lab-Rasool/TCGA", "radiology", split="remedis")84radiology_radimagenet = load_dataset("Lab-Rasool/TCGA", "radiology", split="radimagenet")85```86 87Example code for loading HF dataset into a PyTorch Dataloader.88**Note**: Some embeddings are stored as buffers due to their multi-dimensional shape.89 90```python91from datasets import load_dataset92import os93from torch.utils.data import Dataset94import numpy as np95 96class CustomDataset(Dataset):97    def __init__(self, hf_dataset):98        self.hf_dataset = hf_dataset99 100    def __len__(self):101        return len(self.hf_dataset)102    103    def __getitem__(self, idx):104        hf_item = self.hf_dataset[idx]105        embedding = np.frombuffer(hf_item["embedding"], dtype=np.float32)106        embedding_shape = hf_item["embedding_shape"]107        embedding = embedding.reshape(embedding_shape)108        return embedding109 110if __name__ == "__main__":111    112    # Load clinical embeddings from different models113    clinical_gatortron = load_dataset("Lab-Rasool/TCGA", "clinical", split="gatortron")114    clinical_llama = load_dataset("Lab-Rasool/TCGA", "clinical", split="llama")115    wsi_dataset = load_dataset("Lab-Rasool/TCGA", "wsi", split="uni")116    117    # Example: Access embeddings118    for index, item in enumerate(clinical_gatortron):119        embedding = np.frombuffer(item.get("embedding"), dtype=np.float32).reshape(item.get("embedding_shape"))120        print(f"GatorTron embedding shape: {embedding.shape}")  # Shape: (1024,)121        break122        123    for index, item in enumerate(clinical_llama):124        embedding = np.frombuffer(item.get("embedding"), dtype=np.float32).reshape(item.get("embedding_shape"))125        print(f"Llama embedding shape: {embedding.shape}")  # Shape: (2304,)126        break127``` 128 129## Dataset Statistics130 131### Clinical Data132- **10,771 patient records** per model133- **113 columns** including clinical metadata and embeddings134- **Embedding dimensions:**135  - GatorTron: 1024136  - MedGemma: 2560137  - Qwen: 1024138  - Llama: 2304139  - BioBERT: 768140 141### Pathology Reports142- **10,857 patient records** per model143- **17 columns** including pathology metadata and embeddings144- **Embedding dimensions:**145  - GatorTron: 1024146  - MedGemma: 2560147  - Qwen: 1024148  - Llama: 2304149 150## Dataset Creation151 152#### Data Collection and Processing153The raw data for this dataset was acquired using MINDS, a multimodal data aggregation tool developed by Lab Rasool. 154The collected data includes clinical information, pathology reports, and whole slide images from The Cancer Genome Atlas (TCGA). 155The embeddings were generated using the HoneyBee embedding processing tool, which utilizes foundational models such as GatorTron, MedGemma, Qwen, Llama, and UNI.156 157#### Who are the source data producers?158The source data for this dataset was originally collected and maintained by The Cancer Genome Atlas (TCGA) program, a landmark cancer genomics project jointly managed by the National Cancer Institute (NCI).159 160 161## Citation162<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->163 164```165@article{honeybee,166      title={HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models}, 167      author={Aakash Tripathi and Asim Waqas and Yasin Yilmaz and Ghulam Rasool},168      year={2024},169      eprint={2405.07460},170      archivePrefix={arXiv},171      primaryClass={cs.LG}172}173@article{waqas2024senmo,174  title={SeNMo: A self-normalizing deep learning model for enhanced multi-omics data analysis in oncology},175  author={Waqas, Asim and Tripathi, Aakash and Ahmed, Sabeen and Mukund, Ashwin and Farooq, Hamza and Schabath, Matthew B and Stewart, Paul and Naeini, Mia and Rasool, Ghulam},176  journal={arXiv preprint arXiv:2405.08226},177  year={2024}178}179```180 181### For more information about the data acquisition and processing tools used in creating this dataset, please refer to the following resources:182 183- MINDS paper: https://pubmed.ncbi.nlm.nih.gov/38475170/184- MINDS codebase: https://github.com/lab-rasool/MINDS185- HoneyBee paper: https://arxiv.org/abs/2405.07460186- HoneyBee codebase: https://github.com/lab-rasool/HoneyBee/187 188## Contact Information189For any questions or issues, please contact the dataset curators at [aakash.tripathi@moffitt.org].190