redis/langcache-reranker-v2-bce

Redis fine-tuned CrossEncoder model for semantic caching on LangCache

This is a Cross Encoder model finetuned from Alibaba-NLP/gte-reranker-modernbert-base on the [LangCache Sentence Pairs (subsets=['all'], train+val=True)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v3) dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for sentence pair classification.

Model Details

Model Description

• —Model Type: Cross Encoder
• —Base model: Alibaba-NLP/gte-reranker-modernbert-base <!-- at revision f7481e6055501a30fb19d090657df9ec1f79ab2c -->
• —Maximum Sequence Length: 8192 tokens
• —Number of Output Labels: 1 label
• —Training Dataset:
• —[LangCache Sentence Pairs (subsets=['all'], train+val=True)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v3)
• —Language: en
• —License: apache-2.0

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Documentation: Cross Encoder Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("redis/langcache-reranker-v2-modernbert-bce-eps0.5")
# Get scores for pairs of texts
pairs = [
    ['The newer Punts are still very much in existence today and race in the same fleets as the older boats .', 'The newer punts are still very much in existence today and run in the same fleets as the older boats .'],
    ['Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .', 'Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .'],
    ['After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .', 'Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .'],
    ['She married Peter Haygarth on 29 May 1964 in Durban . Her second marriage , to Robin Osborne , took place in 1977 .', 'She married Robin Osborne on May 29 , 1964 in Durban , and her second marriage with Peter Haygarth took place in 1977 .'],
    ['In 2005 she moved to Norway , settled in Geilo and worked as a rafting guide , in 2006 she started mountain biking - races .', 'In 2005 , she moved to Geilo , settling in Norway and worked as a rafting guide . She started mountain bike races in 2006 .'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'The newer Punts are still very much in existence today and race in the same fleets as the older boats .',
    [
        'The newer punts are still very much in existence today and run in the same fleets as the older boats .',
        'Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .',
        'Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .',
        'She married Robin Osborne on May 29 , 1964 in Durban , and her second marriage with Peter Haygarth took place in 1977 .',
        'In 2005 , she moved to Geilo , settling in Norway and worked as a rafting guide . She started mountain bike races in 2006 .',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Cross Encoder Classification

• —Dataset: test_cls
• —Evaluated with <code>CrossEncoderClassificationEvaluator</code>

Cross Encoder Reranking

• —Datasets: NanoQuoraRetrieval_R25, NanoMSMARCO_R25 and NanoNQ_R25
• —Evaluated with <code>CrossEncoderRerankingEvaluator</code> with these parameters:

  {
      "at_k": 1,
      "always_rerank_positives": true
  }

Cross Encoder Nano BEIR

• —Dataset: NanoBEIR_R25_mean
• —Evaluated with <code>CrossEncoderNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "QuoraRetrieval",
          "MSMARCO",
          "NQ"
      ],
      "rerank_k": 25,
      "at_k": 1,
      "always_rerank_positives": true
  }

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

LangCache Sentence Pairs (subsets=['all'], train+val=True)

• —Dataset: [LangCache Sentence Pairs (subsets=['all'], train+val=True)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v3)
• —Size: 40,015,318 training samples
• —Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:-------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------| | type | string | string | float | | details | <ul><li>min: 27 characters</li><li>mean: 110.99 characters</li><li>max: 199 characters</li></ul> | <ul><li>min: 29 characters</li><li>mean: 110.74 characters</li><li>max: 191 characters</li></ul> | <ul><li>min: 0.25</li><li>mean: 0.49</li><li>max: 0.75</li></ul> |
• —Samples: | sentence1 | sentence2 | label | |:----------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------| | <code>The film was remade in Telugu with the same name in 1981 by Chandra Mohan , Jayasudha , Chakravarthy and S. P. Balasubrahmanyam starring K. Vasu .</code> | <code>The film was written in Telugu with the same name in 1981 by Chandra Mohan , Jayasudha , Chakravarthy and S. P. Balasubrahmanyam with K. Vasu .</code> | <code>0.75</code> | | <code>The Cauchy stress tensor has therefore the following form</code> | <code>Therefore , the cauchy tensor has the following form</code> | <code>0.75</code> | | <code>John Jairo Culma , sometimes spelled as Jhon Jairo Culma ( born 17 March 1981 ) is a Colombian footballer .</code> | <code>John Jairo Culma , sometimes known as Jhon Jairo Culma ( born March 17 , 1981 ) , is a Colombian footballer .</code> | <code>0.75</code> |
• —Loss: <code>BinaryCrossEntropyLoss</code> with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": 0.1028856709599495
  }

Evaluation Dataset

LangCache Sentence Pairs (split=test)

• —Dataset: LangCache Sentence Pairs (split=test)
• —Size: 74,265 evaluation samples
• —Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:-------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | <ul><li>min: 27 characters</li><li>mean: 112.72 characters</li><li>max: 197 characters</li></ul> | <ul><li>min: 27 characters</li><li>mean: 112.54 characters</li><li>max: 198 characters</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> |
• —Samples: | sentence1 | sentence2 | label | |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code> | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code> | <code>1</code> | | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> | | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> |
• —Loss: <code>BinaryCrossEntropyLoss</code> with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": 0.1028856709599495
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 48
• —per_device_eval_batch_size: 48
• —learning_rate: 0.0002
• —weight_decay: 0.001
• —num_train_epochs: 5
• —warmup_ratio: 0.1
• —load_best_model_at_end: True
• —optim: adamw_torch
• —ddp_find_unused_parameters: False
• —push_to_hub: True
• —hub_model_id: redis/langcache-reranker-v2-modernbert-bce-eps0.5
• —eval_on_start: True
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 48
• —per_device_eval_batch_size: 48
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 0.0002
• —weight_decay: 0.001
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 5
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: True
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: False
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: True
• —resume_from_checkpoint: None
• —hub_model_id: redis/langcache-reranker-v2-modernbert-bce-eps0.5
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: True
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: True
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>