lak882/open-clinical-cases-embeddings

PubMed Clinical Cases with SapBERT Embeddings

This dataset contains clinical case narratives extracted from PubMed, processed and enriched with SapBERT embeddings. Each entry represents a clinical scenario, providing a rich source of biomedical text data with associated vector representations.

The embeddings were generated using the cambridgeltl/SapBERT-from-PubMedBERT-fulltext model, a powerful biomedical text encoder, applied to the text field of the original dataset.

Dataset Structure

The dataset is provided as a Parquet file and contains the following columns:

• —text: The original clinical case narrative from PubMed.
• —subfield: (If available in the original dataset) The medical subfield associated with the case.
• —embedding: A 768-dimensional vector embedding generated by SapBERT for the corresponding text.

Dataset Creation

The dataset was created by:

1. 1.Loading clinical case narratives from the rntc/open-clinical-cases-pubmed dataset on Hugging Face.
2. 2.Preprocessing the text data, including handling missing values and removing duplicates.
3. 3.Generating embeddings for the text field using the cambridgeltl/SapBERT-from-PubMedBERT-fulltext model.
4. 4.Combining the original data with the generated embeddings.
5. 5.Saving the resulting dataset as a Parquet file.

Usage

This dataset can be used for various biomedical natural language processing tasks, such as:

• —Semantic search and retrieval of clinical cases.
• —Clustering and analysis of clinical case narratives.
• —Training or fine-tuning downstream models for tasks like clinical entity recognition, relation extraction, or question answering.

Licensing

The licensing of this dataset is based on the original source data from PubMed and the rntc/open-clinical-cases-pubmed dataset. Please refer to the original sources for specific licensing information.

Citation

Please cite the original PubMed sources and the SapBERT model if you use this dataset.