pankajrajdeo/Bioformer-8L-UMLS-Pubmed_PMC-Backward_TCE-Epoch-1

SentenceTransformer

This is a sentence-transformers model trained on the parquet dataset. It maps sentences & paragraphs to a 512-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer <!-- - Base model: Unknown -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 512 dimensions
• —Similarity Function: Cosine Similarity
• —Training Dataset:
• —parquet <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 512, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pankajrajdeo/Bioformer-8L-UMLS-Pubmed_PMC-Backward_TCE-Epoch-1")
# Run inference
sentences = [
    'Stage-specific protein synthesis by isolated spermatogenic cells throughout meiosis and early spermiogenesis in the mouse.',
    'Spermatogenic cells isolated from prepubertal and adult mice by unit gravity sedimentation have been used to examine proteins synthesized in a stage-specific manner throughout meiosis and early spermiogenesis. Preleptotene, leptotene/zygotene, and pachytene spermatocytes were isolated from 17-day-old mice. Adult pachytene spermatocytes and round spermatids were isolated from mature animals. These germ cells were then cultured in defined medium withmethioninemet, when expressed either as cpm/-10(6) cells or cpm/mg protein. Comparisons of 2D autoradiograms indicated that many proteins, including actin and tubulins, are synthesized at approximately equal levels in all stages examined. Other proteins, including heat-shock proteins and multiple plasma membrane constituents, are synthesized in a stage-specific manner in leptotene/zygotene spermatocytes, pachytene spermatocytes, and round spermatids. These studies define conditions for monitoring protein synthesis in isolated spermatogenic cells prior to the pachytene stage of meiosis, provide a 2D map of proteins synthesized at these earlier meiotic stages, and examine the synthesis of several proteins previously identified on 2D gels with biochemical and immunological methods.',
    'Valine-derived benzoxazinones have been synthesized and found to be competitive, slow-binding inhibitors of human leukocyte elastase (HLE). Steady-state inhibition constants Ki are dependent on aryl substitution and reach a maximum of potency of 0.5 nM with the 5-Cl compound 6. UV-spectral data for the interaction of HLE and the unsubstituted inhibitor 3 indicate that the stable complex formed between enzyme and inhibitor is an acyl-enzyme that can either undergo ring closure, to reform intact benzoxazinone, or hydrolysis, to liberate an N-acylanthranilic acid. "Burst" kinetic data, derived from the direct observation of the interaction of HLE and 3, are consistent with results of the inhibition of catalysis experiments.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

parquet

• —Dataset: parquet
• —Size: 33,870,508 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 5 tokens</li><li>mean: 24.6 tokens</li><li>max: 68 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 270.2 tokens</li><li>max: 512 tokens</li></ul> |
• —Samples: | anchor | positive | |:-------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Underlying synovial sarcoma undiagnosed for more than 20 years in a patient with regional pain: a case report.</code> | <code>Synovial sarcoma (SS) is a malignant tumor comprising 5-10% of all soft tissue sarcomas. SS has distinct characteristics, such as a predilection for young adults and relatively slow growth compared to other soft tissue sarcomas. Some patients with SS experience long-standing pain at the tumor site before the development of a palpable mass. Herein, we report the case of a 39-year-old woman with SS in the upper arm who presented with pain for > 20 years. The tumor detected on magnetic resonance imaging at 17 years was an SS. To the best of our knowledge, no English-language reports on imaging study-based identification of SS, which was undiagnosed for > 20 years, are known in the literature. This report discusses the imaging features of this latent lesion and the volume-doubling time of this unusual tumor.</code> | | <code>MR imaging signs of shoulder adhesive capsulitis: analysis of potential differentials and improved diagnostic criteria.</code> | <code>OBJECTIVE: To evaluate the prevalence of shoulder adhesive capsulitis. Based on these findings, a grading system for fibro-inflammatory capsular changes is proposed. CONCLUSION: MR AC signs are frequent in patients with shoulder conditions other than AC; however, in these patients, capsular changes are less prominent than in patients with clinical AC.</code> | | <code>MR imaging signs of shoulder adhesive capsulitis: analysis of potential differentials and improved diagnostic criteria.</code> | <code>OBJECTIVE: To evaluate the prevalence of shoulder adhesive capsulitis. Based on these findings, a grading system for fibro-inflammatory capsular changes is proposed. CONCLUSION: MR AC signs are frequent in patients with shoulder conditions other than AC; however, in these patients, capsular changes are less prominent than in patients with clinical AC.</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }

Evaluation Dataset

parquet

• —Dataset: parquet
• —Size: 33,870,508 evaluation samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 6 tokens</li><li>mean: 18.94 tokens</li><li>max: 60 tokens</li></ul> | <ul><li>min: 26 tokens</li><li>mean: 199.08 tokens</li><li>max: 512 tokens</li></ul> |
• —Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Effect of ritodrine on thyroid hormone concentrations.</code> | <code>In a clinical study of 17 pregnant women treated with ritodrine, a beta-2-sympathomimetic agent used for tocolysis, thyroid hormone status was assessed longitudinally. This was done in order to verify the hypothesis that an increase in T3 levels could result from adrenergic stimulation, since propanolol, a beta blocking agent, has proved to decrease T3 levels in man. We have observed a significant increase in serum T3 concentrations 24-48 h after the start of the ritodrine treatment. The changes were only temporarely since one week after the start the serum T3 concentrations did not differ significantly from the pre-treatment levels. A decrease in T3 levels was found after discontinuation of treatment. No significant changes were found in T4 and TSH concentrations excluding an influence in ritodrine therapy on the pituitary-thyroid axes. It was concluded that stimulation of type I deiodinase was responsible for the changes in T3. These beta-2-mimetic variations may explain, to a certai...</code> | | <code>beta-Aminoisobutyric acid as a marker of thymine catabolism in malignancy.</code> | <code>Urine from untreated patients with various tumours and controls has been examined for the excretion of beta-aminoisobutyric acid and uric acid. The patients were classified into four groups: I, beta-aminoisobutyric acid and uric acid both normal; II, beta-aminoisobutyric acid normal, uric acid elevated; III, beta-aminoisobutyric acid elevated, uric acid normal; IV, beta-aminoisobutyric acid and uric acid both elevated. Uric acid was used as an indicator for tissue-breakdown. Pseudouridine being a specific parameter for t-RNA degradation was estimated for comparison. Increased urinary concentrations of beta-aminoisobutyric acid were frequently found in tumour patients, especially in patients with leukaemia and non-Hodgkin lymphoma. Tissue breakdown being the cause of the beta-aminoisobutyric aciduria could only be considered in part of the patients. Moreover, strongly elevated ratios of beta-aminoisobutyric acid to uric acid were found. Urinary patterns of pyrimidines and purines were d...</code> | | <code>The effect of oral contraceptives on plasma-free and salivary cortisol and cortisone.</code> | <code>The effect of a low estrogen oral contraceptive (OC) on glucocorticoid levels in plasma and saliva as well as glucocorticoid binding was studied in 23 healthy women using 30 micrograms ethinyl estradiol (EE2) + 150 micrograms desogestrel (Marvelon) (II). Fifteen healthy females with normal menses served as controls (I). Blood and salivary samples were taken between 9.00 and 9.30 a.m. on the 18th day of menstrual or pill cycle. Assay accuracy had been optimised by applying extraction and chromatographic purification before radioimmunoassay (RIA) of cortisol and cortisone in both plasma and salivary samples. Free steroid assays were performed by applying the same procedure to equilibrium dialysates obtained after dialysing plasma against an equal volume of buffer, instead of measuring tracer distribution. Corticosteroid Binding Globulin (CBG) was measured by a commercial RIA. As expected, CBG as well as plasma total cortisol were elevated in the pill group. Interestingly both plasma free...</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 128
• —learning_rate: 2e-05
• —num_train_epochs: 1
• —max_steps: 251382
• —log_level: info
• —fp16: True
• —dataloader_num_workers: 16
• —load_best_model_at_end: True
• —resume_from_checkpoint: True

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 128
• —per_device_eval_batch_size: 8
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 2e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 1
• —max_steps: 251382
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.0
• —warmup_steps: 0
• —log_level: info
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 16
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: True
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>

</details>

Framework Versions

• —Python: 3.11.11
• —Sentence Transformers: 3.4.1
• —Transformers: 4.48.2
• —PyTorch: 2.6.0+cu124
• —Accelerate: 1.3.0
• —Datasets: 3.2.0
• —Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->