yosriku/gemma300m-faq_v2

SentenceTransformer

This model was finetuned with Unsloth.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/> based on unsloth/embeddinggemma-300m

This is a sentence-transformers model finetuned from unsloth/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: unsloth/embeddinggemma-300m <!-- at revision bfa3c846ac738e62aa61806ef9112d34acb1dc5a -->
• —Maximum Sequence Length: 1024 tokens
• —Output Dimensionality: 768 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'PeftModelForFeatureExtraction'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "Mau tes buta warna, bisa ke dr. Devi Azri Wahyuni di Siloam Sriwijaya Palembang?",
]
documents = [
    'Berikut informasi profil dokter dr. Devi Azri Wahyuni, SpM (K) dengan spesialisasi Oftalmologi (Mata): Sub-spesialisasi: Subspesialis Neurooftalmologi, Rumah Sakit: Siloam Sriwijaya Palembang, Treatment: Pemeriksaan Mata, Pemeriksaan Slit Lamp, Tes Visus Mata, Pemeriksaan Lapang Pandang, Tonometri, Tes Buta Warna, Gejala/Penyakit: Neuritis Optik, Papilledema, Penyakit Graves, Penyakit Tiroid Mata, Buta Warna, Neuropati Diabetik, Penyakit Mata dan Masalah Penglihatan, Pendidikan: Universitas Respati Indonesia, Kolegium Oftalmologi Indonesia, Universitas Gadjah Mada, Universitas Sriwijaya, Universitas Sriwijaya',
    'Berikut informasi profil dokter Adventia Emilia Krysna Sipi Seda, M.M., M.Psi., Psikolog dengan spesialisasi Psikologi: Sub-spesialisasi: Psikolog, Rumah Sakit: Siloam Hospitals Yogyakarta, Treatment: Manajemen Stres dan Kesehatan Emosional, Gejala/Penyakit: Gangguan Kecemasan, Pendidikan: Universitas Katolik Atma Jaya, Universitas Sanata Dharma',
    'Apa itu Arteriosklerosis? Perbedaan Arteriosklerosis dan Aterosklerosis Penyebab Arteriosklerosis Gejala Arteriosklerosis Diagnosis Arteriosklerosis Pengobatan Arteriosklerosis Komplikasi Arteriosklerosis Pencegahan Arteriosklerosis Arteriosklerosis adalah kondisi medis yang terjadi ketika pembuluh darah arteri mengeras, menebal, dan kehilangan elastisitas karena adanya penumpukan plak lemak pada dinding pembuluh darah. Kondisi ini perlu diwaspadai karena dapat menghambat aliran darah menuju ber',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4944, 0.0387, 0.0981]], dtype=torch.float16)

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 3,281 training samples
• —Columns: <code>question</code> and <code>passage_text</code>
• —Approximate statistics based on the first 1000 samples: | | question | passage_text | |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 7 tokens</li><li>mean: 16.85 tokens</li><li>max: 39 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 223.11 tokens</li><li>max: 1024 tokens</li></ul> |
• —Samples: | question | passage_text | |:-------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>dr A. A. Ayu Nancy Karang itu spesialis apa?</code> | <code>Berikut informasi profil dokter dr. A. A. Ayu Nancy Karang, SpTHT-KL dengan spesialisasi Otorinolaringologi (THTBKL): Sub-spesialisasi: Spesialis Telinga Hidung Tenggorok Bedah Kepala dan Leher, Rumah Sakit: Siloam Hospitals Denpasar, Treatment: Irigasi Nasal, Nasal Endoskopi, Esofagektomi, Tiroidektomi, Gejala/Penyakit: Radang Amandel, Sinusitis, Vertigo, Suara Serak, Rhinitis, Kanker Laring, Polip Hidung, Otitis Media Akut, Pendidikan: Universitas Udayana, Universitas Udayana</code> | | <code>dr. Karina Anindita Sp.PD itu dokter penyakit dalam di Siloam Mataram ya? bisa obatin diabetes?</code> | <code>Berikut informasi profil dokter dr. Karina Anindita, M.Biomed, Sp.PD dengan spesialisasi Penyakit Dalam: Sub-spesialisasi: Spesialis Penyakit Dalam, Rumah Sakit: Siloam Hospitals Mataram, Treatment: Ultrasonografi USG, Computed Tomography CT Scan, Elektrokardiogram EKG, Gejala/Penyakit: Gastritis, Infeksi Saluran Pencernaan, Hipertensi, Hiperkolesterolemia, Diabetes Melitus Tipe 2, Penyakit Autoimun, Asam Urat, Infeksi Saluran Kemih ISK, Infeksi Virus, Pendidikan: Universitas Udayana, Universitas Katolik Atma Jaya</code> | | <code>isk sama mens itu ada hubungannya ga sih? trus biar ga kena isk pas mens gimana?</code> | <code>Apakah Infeksi Saluran Kemih Bisa Menghambat Menstruasi? Apakah Antibiotik Memengaruhi Siklus Menstruasi? Apakah Menstruasi Bisa Meningkatkan Risiko Infeksi Saluran Kemih? Cara Mengurangi Risiko Infeksi Saluran Kemih saat Menstruasi Infeksi saluran kemih (ISK) dikenal sebagai salah satu masalah kesehatan yang cukup umum terjadi, terutama pada wanita. Kondisi ini ditandai dengan adanya gejala seperti nyeri saat buang air kecil, sering buang air kecil, hingga rasa tidak nyaman di bagian perut bawa</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Evaluation Dataset

Unnamed Dataset

• —Size: 411 evaluation samples
• —Columns: <code>question</code> and <code>passage_text</code>
• —Approximate statistics based on the first 411 samples: | | question | passage_text | |:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 7 tokens</li><li>mean: 16.8 tokens</li><li>max: 34 tokens</li></ul> | <ul><li>min: 18 tokens</li><li>mean: 214.26 tokens</li><li>max: 1024 tokens</li></ul> |
• —Samples: | question | passage_text | |:--------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>gerd bisa gara2 stres ga sih?</code> | <code>GERD karena Stres, Bisakah Terjadi? Pengaruh Stres Terhadap Sistem Pencernaan Cara Mengatasi GERD Karena Stres Lambung merupakan organ pencernaan yang memiliki lingkungan relatif asam dengan tingkat keasaman (pH) kurang dari 4.0. Produksi asam lambung memiliki peran penting dalam pencernaan dan pemecahan komponen nutrisi yang dapat diserap oleh tubuh, seperti karbohidrat, asam amino, dan lemak. Ketika mengalami GERD, isi lambung dapat naik kembali ke kerongkongan sehingga menyebabkan gejala yang</code> | | <code>Sosmed itu ngaruh gak sih ke mental remaja? Ada tips biar gak kecanduan?</code> | <code>Pengaruh Media Sosial terhadap Kesehatan Mental Remaja A. Dampak Positif Media Sosial B. Dampak Negatif Media Sosial Tips Mencegah Kecanduan Sosial Media pada Remaja Penggunaan media sosial telah menjadi bagian tak terpisahkan dari kehidupan remaja. Namun, di balik keterhubungannya yang luas, terdapat dampak yang signifikan terhadap kesehatan mental mereka. Bila tidak menggunakannya secara bijak, media sosial dapat meningkatkan risiko cemas dan rendah diri. Mari simak ulasan selengkapnya tentang</code> | | <code>bundle branch block itu apa ya?</code> | <code>Apa itu Bundle Branch Block? Penyebab Bundle Branch Block Penyebab Right Bundle Branch Block Penyebab Left Bundle Branch Block Gejala Bundle Branch Block Diagnosis Bundle Branch Block Pengobatan Bundle Branch Block Bundle branch block (BBB) adalah salah satu gangguan pada impuls atau sinyal listrik yang bertugas untuk membuat bilik jantung (ventrikel) berkontraksi. Kondisi ini dapat menyebabkan detak jantung menjadi tidak teratur dan terkadang bisa menyulitkan kerja jantung dalam memompa darah k</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 64
• —gradient_accumulation_steps: 2
• —learning_rate: 2e-05
• —max_steps: 30
• —warmup_ratio: 0.03
• —prompts: {'question': '', 'passage_text': ''}
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 64
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 2
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 2e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3.0
• —max_steps: 30
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.03
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: {'question': '', 'passage_text': ''}
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• —Python: 3.12.12
• —Sentence Transformers: 5.2.1
• —Transformers: 4.56.2
• —PyTorch: 2.9.0+cu126
• —Accelerate: 1.12.0
• —Datasets: 4.3.0
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->