cyberbabooshka/mtebai

SentenceTransformer based on mixedbread-ai/mxbai-embed-large-v1

This is a sentence-transformers model finetuned from mixedbread-ai/mxbai-embed-large-v1. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: mixedbread-ai/mxbai-embed-large-v1 <!-- at revision db9d1fe0f31addb4978201b2bf3e577f3f8900d2 -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cyberbabooshka/mtebai")
# Run inference
sentences = [
    'What is the relationship between the smallest perturbation of a matrix and its rank, as established in theorems regarding matrix perturbations?',
    '"Suppose $A \\in C^{m \\times n}$ has full column rank (= n). Then $\\min _{\\Delta \\in \\mathbb{C}^{m \\times n}}\\left\\{\\|\\Delta\\|_{2} \\mid A+\\Delta \\text { has rank }<n\\right\\}=\\sigma_{n}(A)$."',
    '"If a beam of light enters and then exits the elevator, the observer on Earth and the one accelerating in empty space must observe the same thing, since they cannot distinguish between being on Earth or accelerating in space. The observer in space, who is accelerating, will observe that the beam of light bends as it crosses the elevator... that means that if the path of a beam of light is curved near Earth, it must be because space itself is curved in the presence of a gravitational field!"',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: eval
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 1,760 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 9 tokens</li><li>mean: 24.87 tokens</li><li>max: 70 tokens</li></ul> | <ul><li>min: 11 tokens</li><li>mean: 68.37 tokens</li><li>max: 500 tokens</li></ul> |
• —Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>How is a proper coloring of a graph defined in the context of vertices and edges?</code> | <code>"A coloring is called proper if for each edge joining two distinct vertices, the two vertices it joins have different colors."</code> | | <code>What is the relationship between the first excited state of the box model and the p orbitals in a hydrogen atom?</code> | <code>"The p orbitals are similar to the first excited state of the box, i.e. $(n{x},n{y},n{z})=(2,1,1)$ is similar to a $p{x}$ orbital, $(n{x},n{y},n{z})=(1,2,1)$ is similar to a $p{y}$ orbital and $(n{x},n{y},n{z})=(1,1,2)$ is similar to a $p{z}$ orbital."</code> | | <code>How can the behavior of the derivative \( f'(x) \) indicate the presence of a local maximum or minimum at a critical point \( x=a \)?</code> | <code>"If there is a local maximum when \( x=a \), the function must be lower near \( x=a \) than it is right at \( x=a \). If the derivative exists near \( x=a \), this means \( f'(x)>0 \) when \( x \) is near \( a \) and \( x < a \), because the function must 'slope up' just to the left of \( a \). Similarly, \( f'(x) < 0 \) when \( x \) is near \( a \) and \( x>a \), because \( f \) slopes down from the local maximum as we move to the right. Using the same reasoning, if there is a local minimum at \( x=a \), the derivative of \( f \) must be negative just to the left of \( a \) and positive just to the right."</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }

Evaluation Dataset

Unnamed Dataset

• —Size: 420 evaluation samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 420 samples: | | anchor | positive | |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 12 tokens</li><li>mean: 24.97 tokens</li><li>max: 66 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 68.52 tokens</li><li>max: 452 tokens</li></ul> |
• —Samples: | anchor | positive | |:------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>What are the two central classes mentioned in the FileSystem framework and what do they represent?</code> | <code>"The class FileReference is the most important entry point to the framework." and "FileSystem is a powerful and elegant library to manipulate files."</code> | | <code>What is the significance of Turing's work in the context of PDE-based models for self-organization of complex systems?</code> | <code>"Turing’s monumental work on the chemical basis of morphogenesis played an important role in igniting researchers’ attention to the PDE-based continuous field models as a mathematical framework to study self-organization of complex systems."</code> | | <code>What are the two options for reducing accelerations as discussed in the passage?</code> | <code>"From the above definitions we see that there are really two options for reducing accelerations. We can reduce the amount that velocity changes, or we can increase the time over which the velocity changes (or both)."</code> |
• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: epoch
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —learning_rate: 2e-05
• —weight_decay: 0.05
• —num_train_epochs: 10
• —warmup_ratio: 0.1
• —fp16: True
• —eval_on_start: True

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: epoch
• —prediction_loss_only: True
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 2e-05
• —weight_decay: 0.05
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 10
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: True
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>

</details>

Framework Versions

• —Python: 3.12.9
• —Sentence Transformers: 4.1.0
• —Transformers: 4.52.3
• —PyTorch: 2.6.0+cu124
• —Accelerate: 1.7.0
• —Datasets: 3.6.0
• —Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->