manojkumaryalaga/medrag-x-pubmedbert

SentenceTransformer based on NeuML/pubmedbert-base-embeddings

This is a sentence-transformers model finetuned from NeuML/pubmedbert-base-embeddings. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: NeuML/pubmedbert-base-embeddings <!-- at revision d6eaca8254bc229f3ca42749a5510ae287eb3486 -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 768 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("manojkumaryalaga/medrag-x-pubmedbert")
# Run inference
sentences = [
    'Do oblique views add value in the diagnosis of spondylolysis in adolescents?',
    'Anteroposterior, lateral, and right and left oblique lumbar spine radiographs are often a standard part of the evaluation of children who are clinically suspected of having spondylolysis. Recent concerns regarding radiation exposure and costs have brought the value of oblique radiographs into question. The purpose of the present study was to determine the diagnostic value of oblique views in the diagnosis of spondylolysis. Radiographs of fifty adolescents with L5 spondylolysis without spondylolisthesis and ',
    "This study sought to compare general surgery research residents' survey information regarding self-efficacy ratings to their observed performance during a simulated small bowel repair. Their observed performance ratings were based on their leadership skills in directing their assistant. Participants were given 15\xa0min to perform a bowel repair using bovine intestines with standardized injuries. Operative assistants were assigned to help assist with the repair. Before the procedure, participants were asked to",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8382, 0.4217],
#         [0.8382, 1.0000, 0.6357],
#         [0.4217, 0.6357, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: pubmed-eval
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 3,000 training samples
• —Columns: <code>sentence0</code> and <code>sentence1</code>
• —Approximate statistics based on the first 1000 samples: | | sentence0 | sentence1 | |:--------|:-----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 8 tokens</li><li>mean: 40.95 tokens</li><li>max: 153 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 88.89 tokens</li><li>max: 141 tokens</li></ul> |
• —Samples: | sentence0 | sentence1 | |:--------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Is low birth weight a risk indicator for congenital cytomegalovirus infection?</code> | <code>Congenital cytomegalovirus infection is currently the leading cause of congenital infection in 0.2-2.2% of live births worldwide leading to variable serious sequalae. The aim of the study was to determine if low birth weight is an indicator of CMV congenital infection evidenced by detecting CMV-DNA in umbilical cord blood at the time of delivery. CMV-IgG and IgM antibodies and CMV-DNAemia were assessed in umbilical cord blood of two hundreds newborns, one hundred of whom had birth weight<or = 2700 gram and/</code> | | <code>Do dermatomyositis and polymyositis affect similar thigh muscles?</code> | <code>Dermatomyositis (DM) and polymyositis (PM) commonly cause weakness of the thigh muscles. However, it is debated whether DM and PM affect similar thigh muscles. Muscle oedema on fat-suppressed MRI sequences is thought to represent active inflammation. In this study, we aimed to assess which thigh muscle groups are preferentially inflamed in DM and PM, respectively, using short-tau inversion-recovery MRI sequences. We analysed 71 patients from 2 Rheumatology centres, 31 with DM and 40 with PM diagnosed accord</code> | | <code>Do older patients receive adequate stroke care?</code> | <code>National guidelines and government directives have adopted policies for urgent assessment of patients with a transient ischaemic attack or minor stroke not admitted to hospital. The risk of recurrent stroke increases substantially with age, as does the potential benefit of secondary prevention. In order to develop effective strategies for older patients, it is important to identify how stroke care is currently provided for this patient group. Between 2004 and 2006, older patients (>75 years) referred to a n</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 32
• —num_train_epochs: 10
• —multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• —do_predict: False
• —eval_strategy: no
• —prediction_loss_only: True
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 32
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1
• —num_train_epochs: 10
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: None
• —warmup_ratio: None
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —enable_jit_checkpoint: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —use_cpu: False
• —seed: 42
• —data_seed: None
• —bf16: False
• —fp16: False
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: -1
• —ddp_backend: None
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —group_by_length: False
• —length_column_name: length
• —project: huggingface
• —trackio_space_id: trackio
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —auto_find_batch_size: False
• —full_determinism: False
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_num_input_tokens_seen: no
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: True
• —use_cache: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: round_robin
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• —Python: 3.12.12
• —Sentence Transformers: 5.2.3
• —Transformers: 5.0.0
• —PyTorch: 2.10.0+cu128
• —Accelerate: 1.12.0
• —Datasets: 4.8.3
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->