cassador/3bs4lr2
068
1---2base_model: indobenchmark/indobert-base-p23datasets:4- afaji/indonli5language:6- id7library_name: sentence-transformers8metrics:9- pearson_cosine10- spearman_cosine11- pearson_manhattan12- spearman_manhattan13- pearson_euclidean14- spearman_euclidean15- pearson_dot16- spearman_dot17- pearson_max18- spearman_max19pipeline_tag: sentence-similarity20tags:21- sentence-transformers22- sentence-similarity23- feature-extraction24- generated_from_trainer25- dataset_size:691526- loss:MultipleNegativesRankingLoss27widget:28- source_sentence: Pesta Olahraga Asia Tenggara atau Southeast Asian Games, biasa29 disingkat SEA Games, adalah ajang olahraga yang diadakan setiap dua tahun dan30 melibatkan 11 negara Asia Tenggara.31 sentences:32 - Sekarang tahun 2017.33 - Warna kulit tidak mempengaruhi waktu berjemur yang baik untuk mengatifkan pro-vitamin34 D3.35 - Pesta Olahraga Asia Tenggara diadakan setiap tahun.36- source_sentence: Menjalani aktivitas Ramadhan di tengah wabah Corona tentunya tidak37 mudah.38 sentences:39 - Tidak ada observasi yang pernah dilansir oleh Business Insider.40 - Wabah Corona membuat aktivitas Ramadhan tidak mudah dijalani.41 - Piala Sudirman pertama digelar pada tahun 1989.42- source_sentence: Dalam bidang politik, partai ini memperjuangkan agar kekuasaan43 sepenuhnya berada di tangan rakyat.44 sentences:45 - Galileo tidak berhasil mengetes hasil dari Hukum Inert.46 - Kudeta 14 Februari 1946 gagal merebut kekuasaan Belanda.47 - Partai ini berusaha agar kekuasaan sepenuhnya berada di tangan rakyat.48- source_sentence: Keluarga mendiang Prince menuduh layanan musik streaming Tidal49 memasukkan karya milik sang penyanyi legendaris tanpa izin .50 sentences:51 - Rosier adalah pelayan setia Lord Voldemort.52 - Bangunan ini digunakan untuk penjualan.53 - Keluarga mendiang Prince sudah memberi izin kepada TImbal untuk menggunakan lagu54 milik Prince.55- source_sentence: Tujuan dari acara dengar pendapat CRTC adalah untuk mengumpulkan56 respons dari pada pemangku kepentingan industri ini dan dari masyarakat umum.57 sentences:58 - Pembuat Rooms hanya bisa membuat meeting yang terbuka.59 - Masyarakat umum dilibatkan untuk memberikan respon dalam acara dengar pendapat60 CRTC.61 - Eminem dirasa tidak akan memulai kembali kariernya tahun ini.62model-index:63- name: SentenceTransformer based on indobenchmark/indobert-base-p264 results:65 - task:66 type: semantic-similarity67 name: Semantic Similarity68 dataset:69 name: IndoNLI dev70 type: IndoNLI-dev71 metrics:72 - type: pearson_cosine73 value: 0.05464572441065177674 name: Pearson Cosine75 - type: spearman_cosine76 value: 0.0581313192236056677 name: Spearman Cosine78 - type: pearson_manhattan79 value: 0.0644062973153787780 name: Pearson Manhattan81 - type: spearman_manhattan82 value: 0.0661721430643920983 name: Spearman Manhattan84 - type: pearson_euclidean85 value: 0.0647291154792417986 name: Pearson Euclidean87 - type: spearman_euclidean88 value: 0.0667018981432360789 name: Spearman Euclidean90 - type: pearson_dot91 value: 0.0214679564614189692 name: Pearson Dot93 - type: spearman_dot94 value: 0.01401560265576529695 name: Spearman Dot96 - type: pearson_max97 value: 0.0647291154792417998 name: Pearson Max99 - type: spearman_max100 value: 0.06670189814323607101 name: Spearman Max102 - task:103 type: semantic-similarity104 name: Semantic Similarity105 dataset:106 name: IndoNLI test107 type: IndoNLI-test108 metrics:109 - type: pearson_cosine110 value: -0.027420454797600895111 name: Pearson Cosine112 - type: spearman_cosine113 value: -0.03327545125556324114 name: Spearman Cosine115 - type: pearson_manhattan116 value: -0.04660713875385687117 name: Pearson Manhattan118 - type: spearman_manhattan119 value: -0.0317801498705458120 name: Spearman Manhattan121 - type: pearson_euclidean122 value: -0.04697128223611728123 name: Pearson Euclidean124 - type: spearman_euclidean125 value: -0.03186507233227842126 name: Spearman Euclidean127 - type: pearson_dot128 value: -0.014150904875791395129 name: Pearson Dot130 - type: spearman_dot131 value: -0.01615774720436149132 name: Spearman Dot133 - type: pearson_max134 value: -0.014150904875791395135 name: Pearson Max136 - type: spearman_max137 value: -0.01615774720436149138 name: Spearman Max139---140 141# SentenceTransformer based on indobenchmark/indobert-base-p2142 143This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2) on the [afaji/indonli](https://huggingface.co/datasets/afaji/indonli) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.144 145## Model Details146 147### Model Description148- **Model Type:** Sentence Transformer149- **Base model:** [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2) <!-- at revision 94b4e0a82081fa57f227fcc2024d1ea89b57ac1f -->150- **Maximum Sequence Length:** 512 tokens151- **Output Dimensionality:** 768 tokens152- **Similarity Function:** Cosine Similarity153- **Training Dataset:**154 - [afaji/indonli](https://huggingface.co/datasets/afaji/indonli)155- **Language:** id156<!-- - **License:** Unknown -->157 158### Model Sources159 160- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)161- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)162- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)163 164### Full Model Architecture165 166```167SentenceTransformer(168 (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 169 (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})170)171```172 173## Usage174 175### Direct Usage (Sentence Transformers)176 177First install the Sentence Transformers library:178 179```bash180pip install -U sentence-transformers181```182 183Then you can load this model and run inference.184```python185from sentence_transformers import SentenceTransformer186 187# Download from the 🤗 Hub188model = SentenceTransformer("cassador/3bs4lr2")189# Run inference190sentences = [191 'Tujuan dari acara dengar pendapat CRTC adalah untuk mengumpulkan respons dari pada pemangku kepentingan industri ini dan dari masyarakat umum.',192 'Masyarakat umum dilibatkan untuk memberikan respon dalam acara dengar pendapat CRTC.',193 'Pembuat Rooms hanya bisa membuat meeting yang terbuka.',194]195embeddings = model.encode(sentences)196print(embeddings.shape)197# [3, 768]198 199# Get the similarity scores for the embeddings200similarities = model.similarity(embeddings, embeddings)201print(similarities.shape)202# [3, 3]203```204 205<!--206### Direct Usage (Transformers)207 208<details><summary>Click to see the direct usage in Transformers</summary>209 210</details>211-->212 213<!--214### Downstream Usage (Sentence Transformers)215 216You can finetune this model on your own dataset.217 218<details><summary>Click to expand</summary>219 220</details>221-->222 223<!--224### Out-of-Scope Use225 226*List how the model may foreseeably be misused and address what users ought not to do with the model.*227-->228 229## Evaluation230 231### Metrics232 233#### Semantic Similarity234* Dataset: `IndoNLI-dev`235* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)236 237| Metric | Value |238|:--------------------|:-----------|239| pearson_cosine | 0.0546 |240| **spearman_cosine** | **0.0581** |241| pearson_manhattan | 0.0644 |242| spearman_manhattan | 0.0662 |243| pearson_euclidean | 0.0647 |244| spearman_euclidean | 0.0667 |245| pearson_dot | 0.0215 |246| spearman_dot | 0.014 |247| pearson_max | 0.0647 |248| spearman_max | 0.0667 |249 250#### Semantic Similarity251* Dataset: `IndoNLI-test`252* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)253 254| Metric | Value |255|:--------------------|:------------|256| pearson_cosine | -0.0274 |257| **spearman_cosine** | **-0.0333** |258| pearson_manhattan | -0.0466 |259| spearman_manhattan | -0.0318 |260| pearson_euclidean | -0.047 |261| spearman_euclidean | -0.0319 |262| pearson_dot | -0.0142 |263| spearman_dot | -0.0162 |264| pearson_max | -0.0142 |265| spearman_max | -0.0162 |266 267<!--268## Bias, Risks and Limitations269 270*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*271-->272 273<!--274### Recommendations275 276*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*277-->278 279## Training Details280 281### Training Dataset282 283#### afaji/indonli284 285* Dataset: [afaji/indonli](https://huggingface.co/datasets/afaji/indonli)286* Size: 6,915 training samples287* Columns: <code>premise</code>, <code>hypothesis</code>, and <code>label</code>288* Approximate statistics based on the first 1000 samples:289 | | premise | hypothesis | label |290 |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|291 | type | string | string | int |292 | details | <ul><li>min: 12 tokens</li><li>mean: 29.26 tokens</li><li>max: 135 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 12.13 tokens</li><li>max: 36 tokens</li></ul> | <ul><li>0: ~51.00%</li><li>1: ~49.00%</li></ul> |293* Samples:294 | premise | hypothesis | label |295 |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------|:---------------|296 | <code>Presiden Joko Widodo (Jokowi) menyampaikan prediksi bahwa wabah virus Corona (COVID-19) di Indonesia akan selesai akhir tahun ini.</code> | <code>Prediksi akhir wabah tidak disampaikan Jokowi.</code> | <code>0</code> |297 | <code>Meski biasanya hanya digunakan di fasilitas kesehatan, saat ini masker dan sarung tangan sekali pakai banyak dipakai di tingkat rumah tangga.</code> | <code>Masker sekali pakai banyak dipakai di tingkat rumah tangga.</code> | <code>1</code> |298 | <code>Seperti namanya, paket internet sahur Telkomsel ini ditujukan bagi pengguna yang menginginkan kuota ekstra, untuk menemani momen sahur sepanjang bulan puasa.</code> | <code>Paket internet sahur tidak ditujukan untuk saat sahur.</code> | <code>0</code> |299* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:300 ```json301 {302 "scale": 20.0,303 "similarity_fct": "cos_sim"304 }305 ```306 307### Evaluation Dataset308 309#### afaji/indonli310 311* Dataset: [afaji/indonli](https://huggingface.co/datasets/afaji/indonli)312* Size: 1,556 evaluation samples313* Columns: <code>premise</code>, <code>hypothesis</code>, and <code>label</code>314* Approximate statistics based on the first 1000 samples:315 | | premise | hypothesis | label |316 |:--------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|317 | type | string | string | int |318 | details | <ul><li>min: 9 tokens</li><li>mean: 28.07 tokens</li><li>max: 179 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 12.15 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>0: ~47.90%</li><li>1: ~52.10%</li></ul> |319* Samples:320 | premise | hypothesis | label |321 |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------|:---------------|322 | <code>Manuskrip tersebut berisi tiga catatan yang menceritakan bagaimana peristiwa jatuhnya meteorit serta laporan kematian akibat kejadian tersebut seperti dilansir dari Science Alert, Sabtu (25/4/2020).</code> | <code>Manuskrip tersebut tidak mencatat laporan kematian.</code> | <code>0</code> |323 | <code>Dilansir dari Business Insider, menurut observasi dari Mauna Loa Observatory di Hawaii pada karbon dioksida (CO2) di level mencapai 410 ppm tidak langsung memberikan efek pada pernapasan, karena tubuh manusia juga masih membutuhkan CO2 dalam kadar tertentu.</code> | <code>Tidak ada observasi yang pernah dilansir oleh Business Insider.</code> | <code>0</code> |324 | <code>Seorang wanita asal New York mengaku sangat benci air putih.</code> | <code>Tidak ada orang dari New York yang membenci air putih.</code> | <code>0</code> |325* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:326 ```json327 {328 "scale": 20.0,329 "similarity_fct": "cos_sim"330 }331 ```332 333### Training Hyperparameters334#### Non-Default Hyperparameters335 336- `eval_strategy`: epoch337- `per_device_train_batch_size`: 4338- `per_device_eval_batch_size`: 4339- `learning_rate`: 2e-05340- `warmup_ratio`: 0.1341- `fp16`: True342 343#### All Hyperparameters344<details><summary>Click to expand</summary>345 346- `overwrite_output_dir`: False347- `do_predict`: False348- `eval_strategy`: epoch349- `prediction_loss_only`: True350- `per_device_train_batch_size`: 4351- `per_device_eval_batch_size`: 4352- `per_gpu_train_batch_size`: None353- `per_gpu_eval_batch_size`: None354- `gradient_accumulation_steps`: 1355- `eval_accumulation_steps`: None356- `learning_rate`: 2e-05357- `weight_decay`: 0.0358- `adam_beta1`: 0.9359- `adam_beta2`: 0.999360- `adam_epsilon`: 1e-08361- `max_grad_norm`: 1.0362- `num_train_epochs`: 3363- `max_steps`: -1364- `lr_scheduler_type`: linear365- `lr_scheduler_kwargs`: {}366- `warmup_ratio`: 0.1367- `warmup_steps`: 0368- `log_level`: passive369- `log_level_replica`: warning370- `log_on_each_node`: True371- `logging_nan_inf_filter`: True372- `save_safetensors`: True373- `save_on_each_node`: False374- `save_only_model`: False375- `restore_callback_states_from_checkpoint`: False376- `no_cuda`: False377- `use_cpu`: False378- `use_mps_device`: False379- `seed`: 42380- `data_seed`: None381- `jit_mode_eval`: False382- `use_ipex`: False383- `bf16`: False384- `fp16`: True385- `fp16_opt_level`: O1386- `half_precision_backend`: auto387- `bf16_full_eval`: False388- `fp16_full_eval`: False389- `tf32`: None390- `local_rank`: 0391- `ddp_backend`: None392- `tpu_num_cores`: None393- `tpu_metrics_debug`: False394- `debug`: []395- `dataloader_drop_last`: False396- `dataloader_num_workers`: 0397- `dataloader_prefetch_factor`: None398- `past_index`: -1399- `disable_tqdm`: False400- `remove_unused_columns`: True401- `label_names`: None402- `load_best_model_at_end`: False403- `ignore_data_skip`: False404- `fsdp`: []405- `fsdp_min_num_params`: 0406- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}407- `fsdp_transformer_layer_cls_to_wrap`: None408- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}409- `deepspeed`: None410- `label_smoothing_factor`: 0.0411- `optim`: adamw_torch412- `optim_args`: None413- `adafactor`: False414- `group_by_length`: False415- `length_column_name`: length416- `ddp_find_unused_parameters`: None417- `ddp_bucket_cap_mb`: None418- `ddp_broadcast_buffers`: False419- `dataloader_pin_memory`: True420- `dataloader_persistent_workers`: False421- `skip_memory_metrics`: True422- `use_legacy_prediction_loop`: False423- `push_to_hub`: False424- `resume_from_checkpoint`: None425- `hub_model_id`: None426- `hub_strategy`: every_save427- `hub_private_repo`: False428- `hub_always_push`: False429- `gradient_checkpointing`: False430- `gradient_checkpointing_kwargs`: None431- `include_inputs_for_metrics`: False432- `eval_do_concat_batches`: True433- `fp16_backend`: auto434- `push_to_hub_model_id`: None435- `push_to_hub_organization`: None436- `mp_parameters`: 437- `auto_find_batch_size`: False438- `full_determinism`: False439- `torchdynamo`: None440- `ray_scope`: last441- `ddp_timeout`: 1800442- `torch_compile`: False443- `torch_compile_backend`: None444- `torch_compile_mode`: None445- `dispatch_batches`: None446- `split_batches`: None447- `include_tokens_per_second`: False448- `include_num_input_tokens_seen`: False449- `neftune_noise_alpha`: None450- `optim_target_modules`: None451- `batch_eval_metrics`: False452- `batch_sampler`: batch_sampler453- `multi_dataset_batch_sampler`: proportional454 455</details>456 457### Training Logs458| Epoch | Step | Training Loss | loss | IndoNLI-dev_spearman_cosine | IndoNLI-test_spearman_cosine |459|:------:|:----:|:-------------:|:------:|:---------------------------:|:----------------------------:|460| 0 | 0 | - | - | 0.1277 | - |461| 0.0578 | 100 | 0.0488 | - | - | - |462| 0.1157 | 200 | 0.0403 | - | - | - |463| 0.1735 | 300 | 0.0173 | - | - | - |464| 0.2313 | 400 | 0.0052 | - | - | - |465| 0.2892 | 500 | 0.0077 | - | - | - |466| 0.3470 | 600 | 0.0065 | - | - | - |467| 0.4049 | 700 | 0.0199 | - | - | - |468| 0.4627 | 800 | 0.0318 | - | - | - |469| 0.5205 | 900 | 0.019 | - | - | - |470| 0.5784 | 1000 | 0.0128 | - | - | - |471| 0.6362 | 1100 | 0.0124 | - | - | - |472| 0.6940 | 1200 | 0.0224 | - | - | - |473| 0.7519 | 1300 | 0.0115 | - | - | - |474| 0.8097 | 1400 | 0.0082 | - | - | - |475| 0.8676 | 1500 | 0.0132 | - | - | - |476| 0.9254 | 1600 | 0.0225 | - | - | - |477| 0.9832 | 1700 | 0.0133 | - | - | - |478| 1.0 | 1729 | - | 0.0173 | 0.0465 | - |479| 1.0411 | 1800 | 0.0056 | - | - | - |480| 1.0989 | 1900 | 0.0027 | - | - | - |481| 1.1567 | 2000 | 0.0109 | - | - | - |482| 1.2146 | 2100 | 0.0021 | - | - | - |483| 1.2724 | 2200 | 0.0004 | - | - | - |484| 1.3302 | 2300 | 0.0082 | - | - | - |485| 1.3881 | 2400 | 0.001 | - | - | - |486| 1.4459 | 2500 | 0.0009 | - | - | - |487| 1.5038 | 2600 | 0.0021 | - | - | - |488| 1.5616 | 2700 | 0.0032 | - | - | - |489| 1.6194 | 2800 | 0.0061 | - | - | - |490| 1.6773 | 2900 | 0.0057 | - | - | - |491| 1.7351 | 3000 | 0.0127 | - | - | - |492| 1.7929 | 3100 | 0.0018 | - | - | - |493| 1.8508 | 3200 | 0.0007 | - | - | - |494| 1.9086 | 3300 | 0.0078 | - | - | - |495| 1.9665 | 3400 | 0.0017 | - | - | - |496| 2.0 | 3458 | - | 0.0078 | 0.0446 | - |497| 2.0243 | 3500 | 0.0003 | - | - | - |498| 2.0821 | 3600 | 0.0042 | - | - | - |499| 2.1400 | 3700 | 0.0005 | - | - | - |500| 2.1978 | 3800 | 0.0002 | - | - | - |501| 2.2556 | 3900 | 0.0006 | - | - | - |502| 2.3135 | 4000 | 0.0003 | - | - | - |503| 2.3713 | 4100 | 0.0048 | - | - | - |504| 2.4291 | 4200 | 0.0002 | - | - | - |505| 2.4870 | 4300 | 0.0043 | - | - | - |506| 2.5448 | 4400 | 0.0011 | - | - | - |507| 2.6027 | 4500 | 0.0005 | - | - | - |508| 2.6605 | 4600 | 0.0009 | - | - | - |509| 2.7183 | 4700 | 0.0013 | - | - | - |510| 2.7762 | 4800 | 0.0018 | - | - | - |511| 2.8340 | 4900 | 0.0004 | - | - | - |512| 2.8918 | 5000 | 0.0014 | - | - | - |513| 2.9497 | 5100 | 0.0045 | - | - | - |514| 3.0 | 5187 | - | 0.0083 | 0.0581 | -0.0333 |515 516 517### Framework Versions518- Python: 3.10.12519- Sentence Transformers: 3.0.1520- Transformers: 4.41.2521- PyTorch: 2.3.0+cu121522- Accelerate: 0.32.1523- Datasets: 2.20.0524- Tokenizers: 0.19.1525 526## Citation527 528### BibTeX529 530#### Sentence Transformers531```bibtex532@inproceedings{reimers-2019-sentence-bert,533 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",534 author = "Reimers, Nils and Gurevych, Iryna",535 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",536 month = "11",537 year = "2019",538 publisher = "Association for Computational Linguistics",539 url = "https://arxiv.org/abs/1908.10084",540}541```542 543#### MultipleNegativesRankingLoss544```bibtex545@misc{henderson2017efficient,546 title={Efficient Natural Language Response Suggestion for Smart Reply}, 547 author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},548 year={2017},549 eprint={1705.00652},550 archivePrefix={arXiv},551 primaryClass={cs.CL}552}553```554 555<!--556## Glossary557 558*Clearly define terms in order to be accessible across audiences.*559-->560 561<!--562## Model Card Authors563 564*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*565-->566 567<!--568## Model Card Contact569 570*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*571-->