Sepideh2027/biolatent-brca-tcga

### Raw Data (`raw/rna/`)

- **Files:** 1,231 original GDC TCGA-BRCA RNA-seq files
- **Format:** STAR gene-count TSV (`*.rna_seq.augmented_star_gene_counts.tsv`)
- **Average file size:** ~4.2 MB

### Processed Data (`processed/`)

| File | Description | Size |
|---|---|---|
| `expression_matrix_log1p.parquet` | ML-ready expression matrix | 248 MB |
| `sample_metadata.csv` | Sample-level metadata | 276 kB |
| `train_samples.csv` | Training split sample IDs | 222 kB |
| `val_samples.csv` | Validation split sample IDs | 26.3 kB |
| `test_samples.csv` | Test split sample IDs | 28.4 kB |

### Expression Matrix Format

The `expression_matrix_log1p.parquet` file is organized as **genes × samples**:

- **Shape:** `23,375 × 1,232` (23,375 genes + 1 `gene_id` column × 1,231 samples + 1 gene ID column)
- **Orientation:** Rows = genes, Columns = samples
- **First column:** `gene_id` (Ensembl gene IDs, e.g., `ENSG00000000003.15`)
- **Remaining columns:** sample UUIDs (matching `sample_metadata.csv` `id` column)
- **Value type:** `float64` — `log1p(TPM)` transformed expression values

### Sample Metadata Columns

- `id`: Internal sample ID (UUID, matches expression matrix column names)
- `file_name`: RNA-seq file name
- `file_id`: GDC file ID
- `case_id`: GDC case ID
- `submitter_id`: TCGA sample barcode (e.g., `TCGA-BH-A18H`)
- `sample_type`: Sample type
- `project_id`: Project ID (`TCGA-BRCA`)

### Sample Type Distribution

| Sample Type | Count | Percentage |
|---|---|---|
| Primary Tumor | 1,111 | 90.3% |
| Solid Tissue Normal | 113 | 9.2% |
| Metastatic | 7 | 0.6% |
| **Total** | **1,231** | **100%** |

## Preprocessing

The `processed/` data was generated from raw GDC RNA-seq files through:

1. Downloaded RNA-seq data from GDC
2. Extracted STAR gene counts
3. Selected protein/gene expression rows
4. Filtered low-expression genes: **`TPM > 1` in at least 10% of samples**
5. Applied `log1p` transformation — i.e., `log(1 + TPM)`

**Note:** No z-score normalization, standardization, batch correction, scaling, PCA, or imputation was applied.

## Train / Validation / Test Splits

The dataset includes predefined splits with **zero overlap** across splits:

| Split | File | Samples | Percentage |
|---|---|---|---|
| Train | `train_samples.csv` | 988 | 80.3% |
| Validation | `val_samples.csv` | 117 | 9.5% |
| Test | `test_samples.csv` | 126 | 10.2% |
| **Total** | | **1,231** | **100%** |

**Overlap verification:** Train ∩ Val = 0, Train ∩ Test = 0, Val ∩ Test = 0.

## Data Quality

- **Samples:** 1,231
- **Genes after QC:** 23,375
- **Transformation:** `log1p(TPM)`
- **Sample types:** Mixed — `Primary Tumor` (90.3%), `Solid Tissue Normal` (9.2%), `Metastatic` (0.6%)
- **Split integrity:** Zero overlap across splits
- **Synthetic:** No

## Intended Use

- Representation learning on transcriptomic data
- Classical ML on cancer gene-expression data
- Research on cancer genomics and precision medicine
- Reproducible computational biology experiments

**Not** intended for clinical diagnosis, treatment decisions, or direct clinical use.

## Limitations

- Derived from a single cancer type (TCGA-BRCA); may not generalize to other cancers without retraining.
- The dataset includes **mixed sample types** (predominantly `Primary Tumor`). Users should filter by `sample_type` if a specific subtype is required.
- Gene filtering applied; the original gene set is larger.
- The expression matrix contains 1,231 samples but the `sample_metadata.csv` may include additional rows — users should verify sample ID matching.
- Not validated for clinical use.

## Licensing and Data Access

The raw data is derived from **TCGA/GDC**, subject to **NIH GDC Data Use Policies**. Redistribution and use are governed by **TCGA/GDC data access terms**, not a generic open-source license.

**Users must comply with GDC data use policies:** https://gdc.cancer.gov/access-data/data-access-policies

## Reproducibility

Related resources:

- **Model:** `Sepideh2027/OmniLatent-TCGA-BRCA-model`
- **Dataset:** `Sepideh2027/biolatent-brca-tcga`

## Citation

If you use this dataset in research, please cite the associated **OmniLatent** research output.

## Contact

**Sepideh Moafi**  
GitHub: [AIResearcher20](https://github.com/AIResearcher20)  
Hugging Face: [Sepideh2027](https://huggingface.co/Sepideh2027)