EMBO/negative_sampling_pmb

1---2license: apache-2.03base_model: answerdotai/ModernBERT-base4tags:5- sentence-transformers6- feature-extraction7- sentence-similarity8- biomedical9- embeddings10- life-sciences11- scientific-text12- SODA-VEC13- EMBO14datasets:15- EMBO/soda-vec-data-full_pmc_title_abstract_paired16metrics:17- cosine-similarity18---19 20# Negative Sampling PMB Model21 22## Model Description23 24SODA-VEC embedding model trained with negative sampling (MultipleNegativesRankingLoss). This is the PMB (PubMed) version, optimized for biomedical text similarity tasks using the standard sentence-transformers approach.25 26This model is part of the **SODA-VEC** (Scientific Open Domain Adaptation for Vector Embeddings) project, which focuses on creating high-quality embedding models for biomedical and life sciences text.27 28**Key Features:**29- Trained on **26.5M biomedical title-abstract pairs** from PubMed Central30- Based on **microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext** architecture31- Optimized for **biomedical text similarity** and **semantic search**32- Produces **768-dimensional embeddings** with mean pooling33 34## Training Details35 36### Training Data37 38- **Dataset**: [`EMBO/soda-vec-data-full_pmc_title_abstract_paired`](https://huggingface.co/datasets/EMBO/soda-vec-data-full_pmc_title_abstract_paired)39- **Size**: 26,473,900 training pairs40- **Source**: Complete PubMed Central baseline (July 2024)41- **Format**: Paired title-abstract examples optimized for contrastive learning42 43### Training Procedure44 45**Loss Function**: MultipleNegativesRankingLoss: standard negative sampling approach used in sentence-transformers46 47**Base Model**: `microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext`48 49**Training Configuration:**50- **GPUs**: 451- **Batch Size per GPU**: 3252- **Gradient Accumulation**: 453- **Effective Batch Size**: 51254- **Learning Rate**: 2e-0555- **Warmup Steps**: 10056- **Pooling Strategy**: mean57- **Epochs**: 1 (full dataset pass)58 59**Training Command:**60```bash61python scripts/soda-vec-train.py --config negative_sampling --push_to_hub --hub_org EMBO --save_limit 562```63 64### Model Architecture65 66- **Base Architecture**: ModernBERT-base (12 layers, 768 hidden size)67- **Pooling**: Mean pooling over token embeddings68- **Output Dimension**: 76869- **Normalization**: L2-normalized embeddings (for VICReg-based models)70 71## Usage72 73### Using Sentence-Transformers74 75```python76from sentence_transformers import SentenceTransformer77 78# Load the model79model = SentenceTransformer("EMBO/negative_sampling_pmb")80 81# Encode sentences82sentences = [83    "CRISPR-Cas9 gene editing in human cells",84    "Genome editing using CRISPR technology"85]86 87embeddings = model.encode(sentences)88print(f"Embedding shape: {embeddings.shape}")89 90# Compute similarity91from sentence_transformers.util import cos_sim92similarity = cos_sim(embeddings[0], embeddings[1])93print(f"Similarity: {similarity.item():.4f}")94```95 96### Using Hugging Face Transformers97 98```python99from transformers import AutoTokenizer, AutoModel100import torch101import torch.nn.functional as F102 103# Load model and tokenizer104tokenizer = AutoTokenizer.from_pretrained("EMBO/negative_sampling_pmb")105model = AutoModel.from_pretrained("EMBO/negative_sampling_pmb")106 107# Encode sentences108sentences = [109    "CRISPR-Cas9 gene editing in human cells",110    "Genome editing using CRISPR technology"111]112 113inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")114with torch.no_grad():115    outputs = model(**inputs)116    117# Mean pooling118embeddings = outputs.last_hidden_state.mean(dim=1)119 120# Normalize (for VICReg models)121embeddings = F.normalize(embeddings, p=2, dim=1)122 123# Compute similarity124similarity = F.cosine_similarity(embeddings[0:1], embeddings[1:2])125print(f"Similarity: {similarity.item():.4f}")126```127 128## Evaluation129 130The model has been evaluated on comprehensive biomedical benchmarks including:131 132- **Journal-Category Classification**: Matching journals to BioRxiv subject categories133- **Title-Abstract Similarity**: Discriminating between related and unrelated paper pairs134- **Field-Specific Separability**: Distinguishing between different biological fields135- **Semantic Search**: Retrieval quality on biomedical text corpora136 137For detailed evaluation results, see the [SODA-VEC benchmark notebooks](https://github.com/source-data/soda-vec).138 139## Intended Use140 141This model is designed for:142 143- **Biomedical Semantic Search**: Finding relevant papers, abstracts, or text passages144- **Scientific Text Similarity**: Computing similarity between biomedical texts145<!-- - **Information Retrieval**: Building search systems for scientific literature146- **Downstream Tasks**: As a base for fine-tuning on specific biomedical tasks147- **Research Applications**: Academic and research use in life sciences148 -->149## Limitations150 151- **Domain Specificity**: Optimized for biomedical and life sciences text; may not perform as well on general domain text152- **Language**: English only153- **Text Length**: Optimized for titles and abstracts; longer documents may require chunking154- **Bias**: Inherits biases from the training data (PubMed Central corpus)155 156## Citation157 158If you use this model, please cite:159 160```bibtex161@software{soda_vec,162  title = {SODA-VEC: Scientific Open Domain Adaptation for Vector Embeddings},163  author = {EMBO},164  year = {2024},165  url = {https://github.com/source-data/soda-vec}166}167```168 169## Model Card Contact170 171For questions or issues, please open an issue on the [SODA-VEC GitHub repository](https://github.com/source-data/soda-vec).172 173---174 175**Model Card Generated**: 2025-11-10176