tomaarsen/reranker-MiniLM-L12-msmarco-scratch-pos_weight-4

CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased using the sentence-transformers library. It computes scores for pairs of texts, which can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Cross Encoder
• —Base model: microsoft/MiniLM-L12-H384-uncased <!-- at revision 44acabbec0ef496f6dbc93adadea57f376b7c0ec -->
• —Maximum Sequence Length: 512 tokens
• —Number of Output Labels: 1 label <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Documentation: Cross Encoder Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("sentence_transformers_model_id")
# Get scores for pairs of texts
pairs = [
    ['enrollment statistics at southern arkansas university', 'The University of Southern Malawi also known as the Malawi University of Science and Technology(MUST) [edit]. The Malawi University of Science and Technology was established on 17th December 2012 by the Malawi University of Science and Technology Act No. 31 of 2012 as the fourth Public University in Malawi.'],
    ['burgos is in what province spain', 'The province of Burgos is a province of northern Spain, in the northeastern part of the autonomous community of Castile and Leon. León it is bordered by the provinces Of, Palencia, Cantabria, Ã\x81lava, Alava álava, La, Rioja, soria Segovia. And valladolid its capital is the City. of burgoshe province of Burgos is divided into 371 municipalities, being the Spanish province with the highest number, although many of them have fewer than 100 inhabitants.'],
    ['most important customer service skills', 'Customer Service Skill #1: Empathy. Empathy gets thrown around a lot in support training, and for good reason: it might be the single most important customer service skill to develop. To help your customers be happy and successful, itâ\x80\x99s important to understand what happiness and success mean to them.'],
    ['what happens if we eat too many carbohydrates', 'What Happens If You Eat Too Many Carbs? We all know the feeling you get after eating a large bowl of pasta. Your stomach swells up and you feel like you just gained 10 pounds. Surprisingly carbohydrates are a very important fuel source for your body. Without them it would be hard to have any energy throughout the day. Even though there are risks to consuming no carbs at all, there are also risks to consuming too much! See the article below where we talk about what could happen if you eat too many carbs. You Will Gain Body Fat Sorry to say this but â\x80\x9cyesâ\x80\x9d if you consume too many carbs than you will gain body fat. This isnâ\x80\x99t all that bad though when it comes to building muscle that is. You need to be eating lots of calories throughout the day in order to spark muscle growth. Carbohydrates just happen to have a lot of calories in them.'],
    ['what county is wharton nj in', 'Sponsored Topics. Wharton is a Borough in Morris County, New Jersey, United States. As of the 2000 United States Census, the borough population was 6,298.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'enrollment statistics at southern arkansas university',
    [
        'The University of Southern Malawi also known as the Malawi University of Science and Technology(MUST) [edit]. The Malawi University of Science and Technology was established on 17th December 2012 by the Malawi University of Science and Technology Act No. 31 of 2012 as the fourth Public University in Malawi.',
        'The province of Burgos is a province of northern Spain, in the northeastern part of the autonomous community of Castile and Leon. León it is bordered by the provinces Of, Palencia, Cantabria, Ã\x81lava, Alava álava, La, Rioja, soria Segovia. And valladolid its capital is the City. of burgoshe province of Burgos is divided into 371 municipalities, being the Spanish province with the highest number, although many of them have fewer than 100 inhabitants.',
        'Customer Service Skill #1: Empathy. Empathy gets thrown around a lot in support training, and for good reason: it might be the single most important customer service skill to develop. To help your customers be happy and successful, itâ\x80\x99s important to understand what happiness and success mean to them.',
        'What Happens If You Eat Too Many Carbs? We all know the feeling you get after eating a large bowl of pasta. Your stomach swells up and you feel like you just gained 10 pounds. Surprisingly carbohydrates are a very important fuel source for your body. Without them it would be hard to have any energy throughout the day. Even though there are risks to consuming no carbs at all, there are also risks to consuming too much! See the article below where we talk about what could happen if you eat too many carbs. You Will Gain Body Fat Sorry to say this but â\x80\x9cyesâ\x80\x9d if you consume too many carbs than you will gain body fat. This isnâ\x80\x99t all that bad though when it comes to building muscle that is. You need to be eating lots of calories throughout the day in order to spark muscle growth. Carbohydrates just happen to have a lot of calories in them.',
        'Sponsored Topics. Wharton is a Borough in Morris County, New Jersey, United States. As of the 2000 United States Census, the borough population was 6,298.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Cross Encoder Reranking

• —Datasets: train-eval, NanoMSMARCO, NanoNFCorpus and NanoNQ
• —Evaluated with <code>CERerankingEvaluator</code>

Cross Encoder Nano BEIR

• —Dataset: NanoBEIR_mean
• —Evaluated with <code>CENanoBEIREvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 2,000,000 training samples
• —Columns: <code>sentence0</code>, <code>sentence1</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | sentence0 | sentence1 | label | |:--------|:-----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | <ul><li>min: 10 characters</li><li>mean: 33.9 characters</li><li>max: 104 characters</li></ul> | <ul><li>min: 64 characters</li><li>mean: 343.08 characters</li><li>max: 991 characters</li></ul> | <ul><li>0: ~81.00%</li><li>1: ~19.00%</li></ul> |
• —Samples: | sentence0 | sentence1 | label | |:-------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | <code>enrollment statistics at southern arkansas university</code> | <code>The University of Southern Malawi also known as the Malawi University of Science and Technology(MUST) [edit]. The Malawi University of Science and Technology was established on 17th December 2012 by the Malawi University of Science and Technology Act No. 31 of 2012 as the fourth Public University in Malawi.</code> | <code>0</code> | | <code>burgos is in what province spain</code> | <code>The province of Burgos is a province of northern Spain, in the northeastern part of the autonomous community of Castile and Leon. León it is bordered by the provinces Of, Palencia, Cantabria, Álava, Alava álava, La, Rioja, soria Segovia. And valladolid its capital is the City. of burgoshe province of Burgos is divided into 371 municipalities, being the Spanish province with the highest number, although many of them have fewer than 100 inhabitants.</code> | <code>1</code> | | <code>most important customer service skills</code> | <code>Customer Service Skill #1: Empathy. Empathy gets thrown around a lot in support training, and for good reason: it might be the single most important customer service skill to develop. To help your customers be happy and successful, it’s important to understand what happiness and success mean to them.</code> | <code>1</code> |
• —Loss: <code>FitMixinLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 64
• —num_train_epochs: 1
• —fp16: True

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 64
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1
• —num_train_epochs: 1
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.0
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• —Energy Consumed: 0.457 kWh
• —Carbon Emitted: 0.178 kg of CO2
• —Hours Used: 1.209 hours

Training Hardware

• —On Cloud: No
• —GPU Model: 1 x NVIDIA GeForce RTX 3090
• —CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• —RAM Size: 31.78 GB

Framework Versions

• —Python: 3.11.6
• —Sentence Transformers: 3.5.0.dev0
• —Transformers: 4.48.3
• —PyTorch: 2.5.0+cu121
• —Accelerate: 1.3.0
• —Datasets: 2.20.0
• —Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->