Modelpublic

swardiantara/bert-tiny-amazon_reviews-k3-fixed-cosine

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes82downloads

Model Card

SentenceTransformer based on google/bertuncasedL-2H-128A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: google/bert_uncased_L-2_H-128_A-2
—Maximum Sequence Length: 128 tokens
—Output Dimensionality: 128 dimensions
—Similarity Function: Cosine Similarity
—Supported Modality: Text

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-amazon_reviews-k3-fixed-cosine")
# Run inference
sentences = [
    "This eyeglasses smells soooo bad. Smell nasty chemical. I would have kept it even if it didn't work for me as other people claimed. I washed them with soap and warm water many times, chemical smell wont go away, so it made me decide to return them.",
    'I ordered two of these. One came broken. It was packaged okay, so I believe it was broken due to the mail carrier tossing it onto my porch. The one that did not break, I love. It looks nice on my counter and serves its purpose. Updated review: Ok so the salt cellar that did not break during shipping pretty much shattered at the lightest touch after 3 weeks on my counter. I can no longer recommend this item, clearly they are not very durable at all.',
    'Broke after one use. Whenever you hit it, the light would go out. Not worth your money.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9934, 0.9923],
#         [0.9934, 1.0000, 0.9955],
#         [0.9923, 0.9955, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

—Size: 200,120 training samples
—Columns: <code>texta</code>, <code>textb</code>, and <code>label</code>
—Approximate statistics based on the first 100 samples: | | texta | textb | label | |:---------|:-----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-----------------------------------| | type | string | string | list | | modality | text | text | | | details | <ul><li>min: 4 tokens</li><li>mean: 40.62 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 23 tokens</li><li>mean: 63.53 tokens</li><li>max: 101 tokens</li></ul> | <ul><li>size: 2 elements</li></ul> |
—Samples: | texta | textb | label | |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------| | <code>Arrived broken. Manufacturer defect. Two of the legs of the base were not completely formed, so there was no way to insert the casters. I unpackaged the entire chair and hardware before noticing this. So, I'll spend twice the amount of time boxing up the whole useless thing and send it back with a 1-star review of part of a chair I never got to sit in. I will go so far as to include a picture of what their injection molding and quality assurance process missed though. I will be hesitant to buy again. It makes me wonder if there aren't missing structures and supports that don't impede the assembly process.</code> | <code>I ordered two of these. One came broken. It was packaged okay, so I believe it was broken due to the mail carrier tossing it onto my porch. The one that did not break, I love. It looks nice on my counter and serves its purpose. Updated review: Ok so the salt cellar that did not break during shipping pretty much shattered at the lightest touch after 3 weeks on my counter. I can no longer recommend this item, clearly they are not very durable at all.</code> | <code>[1.0, 0.0]</code> | | <code>the cabinet dot were all detached from backing... got me</code> | <code>Shipping was late box came open and missing pieces. Very bad experience. Came dirty and a stain on umbrella, not balanced at all so if you dont get weights it will not stand at all.</code> | <code>[1.0, 0.0]</code> | | <code>I received my first order of this product and it was broke so I ordered it again. The second one was broke in more places than the first. I can't blame the shipping process as it's shrink wrapped and boxed.</code> | <code>I ordered two of these. One came broken. It was packaged okay, so I believe it was broken due to the mail carrier tossing it onto my porch. The one that did not break, I love. It looks nice on my counter and serves its purpose. Updated review: Ok so the salt cellar that did not break during shipping pretty much shattered at the lightest touch after 3 weeks on my counter. I can no longer recommend this item, clearly they are not very durable at all.</code> | <code>[1.0, 0.0]</code> |
—Loss: <code>_main_.OrdinalProxyContrastiveLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

—per_device_train_batch_size: 1024
—num_train_epochs: 10
—learning_rate: 2e-05
—load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

—per_device_train_batch_size: 1024
—num_train_epochs: 10
—max_steps: -1
—learning_rate: 2e-05
—lr_scheduler_type: linear
—lr_scheduler_kwargs: None
—warmup_steps: 0
—optim: adamw_torch
—optim_args: None
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—optim_target_modules: None
—gradient_accumulation_steps: 1
—average_tokens_across_devices: True
—max_grad_norm: 1.0
—label_smoothing_factor: 0.0
—bf16: False
—fp16: False
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—use_liger_kernel: False
—liger_kernel_config: None
—use_cache: False
—neftune_noise_alpha: None
—torch_empty_cache_steps: None
—auto_find_batch_size: False
—log_on_each_node: True
—logging_nan_inf_filter: True
—include_num_input_tokens_seen: no
—log_level: passive
—log_level_replica: warning
—disable_tqdm: False
—project: huggingface
—trackio_space_id: None
—trackio_bucket_id: None
—trackio_static_space_id: None
—per_device_eval_batch_size: 8
—prediction_loss_only: True
—eval_on_start: False
—eval_do_concat_batches: True
—eval_use_gather_object: False
—eval_accumulation_steps: None
—include_for_metrics: []
—batch_eval_metrics: False
—save_only_model: False
—save_on_each_node: False
—enable_jit_checkpoint: False
—push_to_hub: False
—hub_private_repo: None
—hub_model_id: None
—hub_strategy: every_save
—hub_always_push: False
—hub_revision: None
—load_best_model_at_end: True
—ignore_data_skip: False
—restore_callback_states_from_checkpoint: False
—full_determinism: False
—seed: 42
—data_seed: None
—use_cpu: False
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—dataloader_prefetch_factor: None
—remove_unused_columns: True
—label_names: None
—train_sampling_strategy: random
—length_column_name: length
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—ddp_static_graph: None
—ddp_backend: None
—ddp_timeout: 1800
—fsdp: None
—fsdp_config: None
—deepspeed: None
—debug: []
—skip_memory_metrics: True
—do_predict: False
—resume_from_checkpoint: None
—warmup_ratio: None
—local_rank: -1
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss
1.0	196	-
2.0	392	-
2.5510	500	0.0016
3.0	588	-
4.0	784	-

—The bold row denotes the saved checkpoint.

Training Time

—Training: 4.0 minutes
—Evaluation: 21.2 seconds
—Total: 4.4 minutes

Framework Versions

—Python: 3.12.4
—Sentence Transformers: 5.5.1
—Transformers: 5.11.0
—PyTorch: 2.5.1+cu121
—Accelerate: 1.13.0
—Datasets: 2.21.0
—Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->