bobox/DeBERTa3-s-CustomPoolin-toytest2-step1

SentenceTransformer based on microsoft/deberta-v3-small

This is a sentence-transformers model finetuned from microsoft/deberta-v3-small. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: microsoft/deberta-v3-small <!-- at revision a36c739020e01763fe789b4b85e2df55d6180012 -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 768 tokens
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DebertaV2Model 
  (1): AdvancedWeightedPooling(
    (alpha_dropout_layer): Dropout(p=0.01, inplace=False)
    (gate_dropout_layer): Dropout(p=0.05, inplace=False)
    (linear_cls_pj): Linear(in_features=768, out_features=768, bias=True)
    (linear_cls_Qpj): Linear(in_features=768, out_features=768, bias=True)
    (linear_mean_pj): Linear(in_features=768, out_features=768, bias=True)
    (linear_attnOut): Linear(in_features=768, out_features=768, bias=True)
    (mha): MultiheadAttention(
      (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
    )
    (layernorm_output): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (layernorm_weightedPooing): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (layernorm_pjCls): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (layernorm_pjMean): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (layernorm_attnOut): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bobox/DeBERTa3-s-CustomPoolin-toytest2-step1")
# Run inference
sentences = [
    'What do outdoor plants require to survive?',
    'a plants require water for survival. If no rain or watering, the plant dies. \n Outdoor plants require rain to survive.',
    "(Vegan) soups are nutritious. In addition to them being easy to digest, most the time, soups are made from nutrient-dense ingredients like herbs, spices, vegetables, and beans. Because the soup is full of those nutrients AND that it's easy to digest, your body is able to absorb more of those nutrients into your system.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Semantic Similarity

• —Dataset: sts-test
• —Evaluated with <code>EmbeddingSimilarityEvaluator</code>

Binary Classification

• —Dataset: allNLI-dev
• —Evaluated with <code>BinaryClassificationEvaluator</code>

Binary Classification

• —Dataset: Qnli-dev
• —Evaluated with <code>BinaryClassificationEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 32,500 training samples
• —Columns: <code>sentence1</code> and <code>sentence2</code>
• —Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 4 tokens</li><li>mean: 29.43 tokens</li><li>max: 400 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 57.02 tokens</li><li>max: 389 tokens</li></ul> |
• —Samples: | sentence1 | sentence2 | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>What is the chemical symbol for Silver?</code> | <code>Chemical Elements.com - Silver (Ag) Bentor, Yinon. Chemical Element.com - Silver. <http://www.chemicalelements.com/elements/ag.html>. For more information about citing online sources, please visit the MLA's Website . This page was created by Yinon Bentor. Use of this web site is restricted by this site's license agreement . Copyright © 1996-2012 Yinon Bentor. All Rights Reserved.</code> | | <code>e.&#9;in solids the atoms are closely locked in position and can only vibrate, in liquids the atoms and molecules are more loosely connected and can collide with and move past one another, while in gases the atoms or molecules are free to move independently, colliding frequently.</code> | <code>Within a substance, atoms that collide frequently and move independently of one another are most likely in a gas</code> | | <code>Keanu Neal was born in 1995 .</code> | <code>Keanu Neal ( born July 26 , 1995 ) is an American football safety for the Atlanta Falcons of the National Football League ( NFL ) .</code> |
• —Loss: <code>GISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.025}

Evaluation Dataset

Unnamed Dataset

• —Size: 1,664 evaluation samples
• —Columns: <code>sentence1</code> and <code>sentence2</code>
• —Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 4 tokens</li><li>mean: 28.9 tokens</li><li>max: 348 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 57.31 tokens</li><li>max: 450 tokens</li></ul> |
• —Samples: | sentence1 | sentence2 | |:--------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Gene expression is regulated primarily at the what level?</code> | <code>Gene expression is regulated primarily at the transcriptional level.</code> | | <code>Diffusion Diffusion is a process where atoms or molecules move from areas of high concentration to areas of low concentration.</code> | <code>Diffusion is the process in which a substance naturally moves from an area of higher to lower concentration.</code> | | <code>In which James Bond film did Sean Connery wear the Bell Rocket Belt (Jet Pack)?</code> | <code>Jet Pack - James Bond Gadgets 125lbs Summary James Bond used the Jetpack in 1965's Thunderball, to escape from gunmen after killing a SPECTRE agent. The Jetpack In the 1965 movie Thunderball, James Bond (Sean Connery) uses Q's Jetpack to escape from two gunmen after killing Jacques Bouvar, SPECTRE Agent No. 6. It was also used in the Thunderball movie posters, being the "Look Up" part of the "Look Up! Look Down! Look Out!" tagline. The Jetpack returned in the 2002 movie Die Another Day, in the Q scene that showcased many other classic gadgets. The Jetpack is a very popular Bond gadget and is a favorite among many fans due to its originality and uniqueness. The Bell Rocket Belt The Jetpack is actually a Bell Rocket Belt, a fully functional rocket pack device. It was designed for use in the army, but was rejected because of its short flying time of 21-22 seconds. Powered by hydrogen peroxide, it could fly about 250m and reach a maximum altitude of 18m, going 55km/h. Despite its impracticality in the real world, the Jetpack made a spectacular debut in Thunderball. Although Sean Connery is seen in the takeoff and landings, the main flight was piloted by Gordon Yeager and Bill Suitor.</code> |
• —Loss: <code>GISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.025}

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 256
• —lr_scheduler_type: cosinewithmin_lr
• —lr_scheduler_kwargs: {'numcycles': 0.5, 'minlr': 3.3333333333333337e-06}
• —warmup_ratio: 0.33
• —save_safetensors: False
• —fp16: True
• —push_to_hub: True
• —hub_model_id: bobox/DeBERTa3-s-CustomPoolin-toytest2-step1-checkpoints-tmp
• —hub_strategy: all_checkpoints
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 256
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3
• —max_steps: -1
• —lr_scheduler_type: cosinewithmin_lr
• —lr_scheduler_kwargs: {'numcycles': 0.5, 'minlr': 3.3333333333333337e-06}
• —warmup_ratio: 0.33
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: True
• —resume_from_checkpoint: None
• —hub_model_id: bobox/DeBERTa3-s-CustomPoolin-toytest2-step1-checkpoints-tmp
• —hub_strategy: all_checkpoints
• —hub_private_repo: False
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —eval_use_gather_object: False
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>