juanmcristobal/e5-small-v2-mitre

SentenceTransformer based on intfloat/e5-small-v2

This is a sentence-transformers model finetuned from intfloat/e5-small-v2 on the parquet dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: intfloat/e5-small-v2 <!-- at revision ffb93f3bd4047442299a41ebb6fa998a38507c52 -->
• —Maximum Sequence Length: 256 tokens
• —Output Dimensionality: 384 dimensions
• —Similarity Function: Cosine Similarity
• —Training Dataset:
• —parquet <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Specifies 32-bit packet size larger than 1048576 bytes causing allocator failure.',
    'Attackers exploit server software weaknesses, exhausting connections or memory to keep users locked out.',
    'Tricking users into opening weaponized attachments triggers code execution via manipulated documents or executables.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 0.9999,  0.7224, -0.0776],
#         [ 0.7224,  1.0000, -0.0670],
#         [-0.0776, -0.0670,  0.9999]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: cti_ir
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

parquet

• —Dataset: parquet
• —Size: 11,538,689 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 4 tokens</li><li>mean: 21.88 tokens</li><li>max: 101 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 20.39 tokens</li><li>max: 46 tokens</li></ul> |
• —Samples: | anchor | positive | |:------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------| | <code>Attackers takeover expired PyPI domain to inject malicious code into developer projects.</code> | <code>Attackers hijack subdomains from non-existent resources to conduct operations under trusted names.</code> | | <code>they initiate contact via Teams, claiming to be from the organization’s help desk.</code> | <code>Threat actors exploit third-party services for spearphishing against specific victims.</code> | | <code>Malicious actors relay emails through compromised servers to obscure origins.</code> | <code>Legitimate web services hijacked en masse become unwitting command-and-control nodes and proxy layers.</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Evaluation Dataset

parquet

• —Dataset: parquet
• —Size: 5,000 evaluation samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 4 tokens</li><li>mean: 21.6 tokens</li><li>max: 83 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 20.66 tokens</li><li>max: 50 tokens</li></ul> |
• —Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------| | <code>The POST request sent by the malware to the Telegram API shown in the ANY.RUN sandbox</code> | <code>Threat actors embed malicious payloads inside normal web traffic patterns like HTTPS or WebSocket exchanges.</code> | | <code>Attackers force website visitors to view shocking imagery accompanied by intimidating messages.</code> | <code>Hackers alter content to undermine integrity with disturbing imagery.</code> | | <code>phishing campaign that targets Office 365 users and includes a convincing SharePoint document claiming to urgently need an email signature.</code> | <code>Spearphishing leverages link-based lures to bypass attachment scrutiny and extract sensitive tokens.</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 32
• —gradient_accumulation_steps: 2
• —learning_rate: 5e-06
• —num_train_epochs: 5
• —warmup_ratio: 0.05
• —bf16: True
• —dataloader_num_workers: 2
• —load_best_model_at_end: True
• —prompts: {'anchor': 'query: ', 'positive': 'passage: '}
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 8
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 2
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-06
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 5
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.05
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: True
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 2
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: {'anchor': 'query: ', 'positive': 'passage: '}
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>