Modelpublic

yasserrmd/pediatrics-gemma-300m-emb

sourceHugging Faceupdated 1y agoView on Hugging Face

1likes55downloads

Model Card

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: google/embeddinggemma-300m
—Maximum Sequence Length: 2048 tokens
—Output Dimensionality: 768 dimensions
—Similarity Function: Cosine Similarity

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("yasserrmd/pediatrics-gemma-300m-emb")
# Run inference
queries = [
    "How can interdisciplinary collaboration and clearly defined strategies help reduce damage and suffering for children and adolescents with mental disorders and their families?\n",
]
documents = [
    'Interdisciplinary collaboration, involving caregivers, mental health managers, and professionals, can play a crucial role in reducing damage and suffering for children and adolescents with mental disorders and their families. By developing a shared vision and practice, professionals can work together to establish dynamic strategies that address the complex demands of these individuals. This collaboration requires critical observation of services, conduct, and strategies to ensure that families and mental health services are brought closer together. By bridging the gap between demands and care, interdisciplinary collaboration can help improve the well-being and outcomes for children, adolescents, and their families.',
    'Pneumothorax in children can have various causes, including lung cysts. In cases where there are signs of pneumothorax without a lung lesion to account for the condition, cysts of the lung should be suspected. The presence of cysts can be confirmed through radiographic evidence, which can also help determine the location and characteristics of the cysts.',
    'The infant in the NICU presented with a degraded general status, intubation, and mechanical ventilation. They also had unilaterally diminished breath sounds, hypoxemia, oliguria, tachycardia, hypotension, abdominal distention, and fever. Additionally, they exhibited hepatosplenomegaly, oliguria progressing to anuria, thrombocytopenia, and hyperchromic hematuria. ',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7673, -0.0352,  0.0221]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

—Size: 20,000 training samples
—Columns: <code>sentence0</code> and <code>sentence1</code>
—Approximate statistics based on the first 1000 samples: | | sentence0 | sentence1 | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 8 tokens</li><li>mean: 20.38 tokens</li><li>max: 64 tokens</li></ul> | <ul><li>min: 17 tokens</li><li>mean: 87.92 tokens</li><li>max: 258 tokens</li></ul> |
—Samples: | sentence0 | sentence1 | |:--------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>What is the role of routine health check-ups in detecting and diagnosing metabolic syndrome and NAFLD in obese children?</code> | <code>Routine health check-ups are important in detecting and diagnosing metabolic syndrome and NAFLD in obese children. However, there is a lack of routine health check-up data specifically for these complications in obese children. To address this need, pediatric health promotion centers and pediatric obesity clinics have been developed. The aim of these centers is to provide routine health check-ups and obesity-oriented check-ups to detect and diagnose metabolic syndrome and NAFLD in children.</code> | | <code>How does the implementation of family-centered rounds (FCR) impact medical education?<br></code> | <code>The implementation of family-centered rounds (FCR) has raised concerns about its potential impact on medical education. Some evidence suggests that FCR may lead to decreased "didactic" teaching, increased discomfort in asking specific management questions, and limited time to discuss management options for residents and students (8, 9, 10, 11). However, the literature on the association between FCR and teaching has mainly focused on learners' perceptions, and there is a lack of objective data to address the relationship between FCR and medical knowledge acquisition.</code> | | <code>What are some common clinical symptoms of neonatal septicaemia?<br></code> | <code>Some common clinical symptoms of neonatal septicaemia include fever, poor feeding, excessive cry, difficulty in breathing, yellowish skin discoloration, skin rashes, jitteriness, and irritability.</code> |
—Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

json

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

—per_device_train_batch_size: 6
—per_device_eval_batch_size: 6
—num_train_epochs: 1
—multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

—overwrite_output_dir: False
—do_predict: False
—eval_strategy: no
—prediction_loss_only: True
—per_device_train_batch_size: 6
—per_device_eval_batch_size: 6
—per_gpu_train_batch_size: None
—per_gpu_eval_batch_size: None
—gradient_accumulation_steps: 1
—eval_accumulation_steps: None
—torch_empty_cache_steps: None
—learning_rate: 5e-05
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—max_grad_norm: 1
—num_train_epochs: 1
—max_steps: -1
—lr_scheduler_type: linear
—lr_scheduler_kwargs: {}
—warmup_ratio: 0.0
—warmup_steps: 0
—log_level: passive
—log_level_replica: warning
—log_on_each_node: True
—logging_nan_inf_filter: True
—save_safetensors: True
—save_on_each_node: False
—save_only_model: False
—restore_callback_states_from_checkpoint: False
—no_cuda: False
—use_cpu: False
—use_mps_device: False
—seed: 42
—data_seed: None
—jit_mode_eval: False
—use_ipex: False
—bf16: False
—fp16: False
—fp16_opt_level: O1
—half_precision_backend: auto
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—local_rank: 0
—ddp_backend: None
—tpu_num_cores: None
—tpu_metrics_debug: False
—debug: []
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_prefetch_factor: None
—past_index: -1
—disable_tqdm: False
—remove_unused_columns: True
—label_names: None
—load_best_model_at_end: False
—ignore_data_skip: False
—fsdp: []
—fsdp_min_num_params: 0
—fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
—fsdp_transformer_layer_cls_to_wrap: None
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—deepspeed: None
—label_smoothing_factor: 0.0
—optim: adamwtorchfused
—optim_args: None
—adafactor: False
—group_by_length: False
—length_column_name: length
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—skip_memory_metrics: True
—use_legacy_prediction_loop: False
—push_to_hub: False
—resume_from_checkpoint: None
—hub_model_id: None
—hub_strategy: every_save
—hub_private_repo: None
—hub_always_push: False
—hub_revision: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—include_inputs_for_metrics: False
—include_for_metrics: []
—eval_do_concat_batches: True
—fp16_backend: auto
—push_to_hub_model_id: None
—push_to_hub_organization: None
—mp_parameters:
—auto_find_batch_size: False
—full_determinism: False
—torchdynamo: None
—ray_scope: last
—ddp_timeout: 1800
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—include_tokens_per_second: False
—include_num_input_tokens_seen: False
—neftune_noise_alpha: None
—optim_target_modules: None
—batch_eval_metrics: False
—eval_on_start: False
—use_liger_kernel: False
—liger_kernel_config: None
—eval_use_gather_object: False
—average_tokens_across_devices: False
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: round_robin
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss
0.1500	500	0.0195
0.2999	1000	0.0095
0.4499	1500	0.0084
0.5999	2000	0.0059
0.7499	2500	0.0021
0.8998	3000	0.0035

Framework Versions

—Python: 3.12.11
—Sentence Transformers: 5.1.0
—Transformers: 4.56.2
—PyTorch: 2.8.0+cu128
—Accelerate: 1.10.1
—Datasets: 4.0.0
—Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->