Modelpublic

yosriku/Indobert-Base-p2-Trash-Medium-EXP2

sourceHugging Faceupdated 9mo agoView on Hugging Face

0likes51downloads

Model Card

SentenceTransformer based on indobenchmark/indobert-base-p2

This is a sentence-transformers model finetuned from indobenchmark/indobert-base-p2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: indobenchmark/indobert-base-p2
—Maximum Sequence Length: 512 tokens
—Output Dimensionality: 768 dimensions
—Similarity Function: Cosine Similarity

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("yosriku/Indobert-Base-p2-Trash-Medium-EXP2")
# Run inference
sentences = [
    'ini, katakan: “Halo,”nya. 4?',
    'Pasal 5 Cukup jelas. Pasal 6 Huruf a Cukup jelas. Huruf b',
    '(4) Setiap orang berhak untuk berperan dalam perlindungan dan pengelolaan lingkungan hidup sesuai dengan peraturan perundang-undangan.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3311, 0.1417],
#         [0.3311, 1.0000, 0.1066],
#         [0.1417, 0.1066, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

—Size: 4,516 training samples
—Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
—Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | string | | details | <ul><li>min: 8 tokens</li><li>mean: 20.04 tokens</li><li>max: 80 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 35.53 tokens</li><li>max: 117 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 35.65 tokens</li><li>max: 117 tokens</li></ul> |
—Samples: | anchor | positive | negative | |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Apa tujuan utama yang ingin dicapai melalui riset penggunaan teknologi gasifikasi di Pantai Parangtritis?</code> | <code>Penelitian ini bertujuan untuk mengetahui besarnya potensi energi listrik yang dihasilkan dari sampah organik d i Kawasan Wisata Pantai Parangtritis menggunakan proses gasifikasi</code> | <code>bahwa dalam pengelolaan sampah diperlukan kepastian hukum, kejelasan tanggung jawab dan kewenangan Pemerintah, pemerintahan daerah, serta peran masyarakat dan dunia usaha sehingga pengelolaan sampah dapat berjalan secara proporsional, efektif, dan</code> | | <code>Jelaskan transparansi pemerintah terkait permohonan dan keputusan izin lingkungan kepada publik.</code> | <code>Pasal 39 (1) Menteri, gubernur, atau bupati/walikota sesuai dengan kewenangannya wajib mengumumkan setiap permohonan dan keputusan izin lingkungan</code> | <code>. bahwa untuk p enanganan sampah laut diperlukan komi tmen b. bahwa akibat pencemaran sampah plastik d i laut, telah ditemuk an k andung an plastik berukuran mikro dan nano pada biota dan surnb er daya laut di perairan Indonesia;</code> | | <code>menjadi ini: Kalau kita ingin mendapatkan pengelolaan fisik sampah laut, maka ubah kalimat itu. Sekarang! mereka-sama? lagi sampah tersebut dibuang jadi</code> | <code>pengelolaan sampah yang bersumber dari darat; c. penanggulangan sampah di pesisir dan laut; d. mekanisme pendanaan, penguatan kelembagaan, pengawasan, dan penegakan hukum;</code> | <code>Pasal 118 Terhadap tindak pidana sebagaimana dimaksud dalam Pasal 116 ayat (1) huruf a, sanksi pidana dijatuhkan kepada badan usaha yang diwakili oleh pengurus yang berwenang mewakili di dalam dan di luar pengadilan sesuai dengan peraturan</code> |
—Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

json

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

—per_device_train_batch_size: 64
—learning_rate: 2e-05
—fp16: True
—push_to_hub: True
—hub_model_id: yosriku/Indobert-Base-p2-Trash-Medium-EXP2
—hub_strategy: end
—hub_private_repo: False

All Hyperparameters

<details><summary>Click to expand</summary>

—overwrite_output_dir: False
—do_predict: False
—eval_strategy: no
—prediction_loss_only: True
—per_device_train_batch_size: 64
—per_device_eval_batch_size: 8
—per_gpu_train_batch_size: None
—per_gpu_eval_batch_size: None
—gradient_accumulation_steps: 1
—eval_accumulation_steps: None
—torch_empty_cache_steps: None
—learning_rate: 2e-05
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—max_grad_norm: 1.0
—num_train_epochs: 3
—max_steps: -1
—lr_scheduler_type: linear
—lr_scheduler_kwargs: {}
—warmup_ratio: 0.0
—warmup_steps: 0
—log_level: passive
—log_level_replica: warning
—log_on_each_node: True
—logging_nan_inf_filter: True
—save_safetensors: True
—save_on_each_node: False
—save_only_model: False
—restore_callback_states_from_checkpoint: False
—no_cuda: False
—use_cpu: False
—use_mps_device: False
—seed: 42
—data_seed: None
—jit_mode_eval: False
—bf16: False
—fp16: True
—fp16_opt_level: O1
—half_precision_backend: auto
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—local_rank: 0
—ddp_backend: None
—tpu_num_cores: None
—tpu_metrics_debug: False
—debug: []
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_prefetch_factor: None
—past_index: -1
—disable_tqdm: False
—remove_unused_columns: True
—label_names: None
—load_best_model_at_end: False
—ignore_data_skip: False
—fsdp: []
—fsdp_min_num_params: 0
—fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
—fsdp_transformer_layer_cls_to_wrap: None
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—deepspeed: None
—label_smoothing_factor: 0.0
—optim: adamwtorchfused
—optim_args: None
—adafactor: False
—group_by_length: False
—length_column_name: length
—project: huggingface
—trackio_space_id: trackio
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—skip_memory_metrics: True
—use_legacy_prediction_loop: False
—push_to_hub: True
—resume_from_checkpoint: None
—hub_model_id: yosriku/Indobert-Base-p2-Trash-Medium-EXP2
—hub_strategy: end
—hub_private_repo: False
—hub_always_push: False
—hub_revision: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—include_inputs_for_metrics: False
—include_for_metrics: []
—eval_do_concat_batches: True
—fp16_backend: auto
—push_to_hub_model_id: None
—push_to_hub_organization: None
—mp_parameters:
—auto_find_batch_size: False
—full_determinism: False
—torchdynamo: None
—ray_scope: last
—ddp_timeout: 1800
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—include_tokens_per_second: False
—include_num_input_tokens_seen: no
—neftune_noise_alpha: None
—optim_target_modules: None
—batch_eval_metrics: False
—eval_on_start: False
—use_liger_kernel: False
—liger_kernel_config: None
—eval_use_gather_object: False
—average_tokens_across_devices: True
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss
0.1408	10	3.4762
0.2817	20	2.9233
0.4225	30	2.6656
0.5634	40	2.7033
0.7042	50	2.8019
0.8451	60	2.7314
0.9859	70	2.7595
1.1268	80	2.3427
1.2676	90	2.2645
1.4085	100	2.2181
1.5493	110	2.0353
1.6901	120	2.1682
1.8310	130	2.1819
1.9718	140	2.0523
2.1127	150	1.8393
2.2535	160	1.7487
2.3944	170	1.9105
2.5352	180	1.8721
2.6761	190	1.7491
2.8169	200	1.9764
2.9577	210	1.8829

Framework Versions

—Python: 3.12.12
—Sentence Transformers: 5.2.0
—Transformers: 4.57.3
—PyTorch: 2.9.0+cu126
—Accelerate: 1.12.0
—Datasets: 4.0.0
—Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->