MinhPhuc0804/me5-docling-checkthat-task1-v1

SentenceTransformer based on intfloat/multilingual-e5-large-instruct

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large-instruct. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: intfloat/multilingual-e5-large-instruct <!-- at revision 274baa43b0e13e37fafa6428dbc7938e62e5c439 -->
• —Maximum Sequence Length: 256 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity
• —Supported Modality: Text <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MinhPhuc0804/me5-docling-checkthat-task1-v1")
# Run inference
sentences = [
    "query: @user You're more prone to get COVID-19 with more of the jabs. They're aiming to cut down Earth's population.",
    'passage: doses. Results Among 51017 employees, COVID-19 occurred in 4424 (8.7%) during the study.\n\ntitle: Effectiveness of the Coronavirus Disease 2019 (COVID-19) Bivalent Vaccine\nIn multivariable analysis, the bivalent vaccinated state was associated with lower risk of COVID-19 during the BA.4/5 dominant (HR, .71; 95% C.I., .63-.79) and the BQ dominant (HR, .80; 95% C.I., .69-.94) phases, but decreased risk was not found during the XBB dominant phase (HR, .96; 95% C.I., .82-.1.12). Estimated vaccine effectiveness (VE) was 29% (95% C.I., 21%-37%), 20% (95% C.I., 6%-31%), and 4% (95% C.I., -12%-18%), during the BA.4/5, BQ, and XBB dominant phases, respectively. Risk of COVID-19 also increased with time since most recent prior COVID-19 episode and with the number of vaccine doses previously received.',
    'passage: d the bacterial counts from the same surgeon, a significant increase was noted in the 2-hours group.\n\ntitle: Surgical masks as source of bacterial contamination during operative procedures\nMoreover, the bacterial counts were significantly higher among the surgeons than the OR. Additionally, the bacterial count of the external surface of the second mask was significantly higher than that of the first one. The source of bacterial contamination in SMs was the body surface of the surgeons rather than the OR environment. Moreover, we recommend that surgeons should change the mask after each operation, especially those beyond 2 hours. Double-layered SMs or those with excellent filtration function may also be a better alternative. This study provides strong evidence for the identification that SMs as source of bacterial contamination during operative procedures, which should be a cause for alarm and attention in the prevention of surgical site infection in clinical practice.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6161, 0.0673],
#         [0.6161, 1.0000, 0.0795],
#         [0.0673, 0.0795, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: 10-percent-dev-split
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 17,319 training samples
• —Columns: <code>sentence0</code> and <code>sentence1</code>
• —Approximate statistics based on the first 1000 samples: | | sentence0 | sentence1 | |:--------|:------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 19 tokens</li><li>mean: 60.83 tokens</li><li>max: 106 tokens</li></ul> | <ul><li>min: 26 tokens</li><li>mean: 184.65 tokens</li><li>max: 256 tokens</li></ul> |
• —Samples: | sentence0 | sentence1 | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>query: Financial ties between heads of powerful US professional health societies and the corporate sector: cross‑sectional study \| The BMJ</code> | <code>passage: title: Financial ties between leaders of influential US professional medical associations and industry: cross sectional study abstract: To investigate the nature and extent of financial relationships between leaders of influential professional medical associations in the United States and pharmaceutical and device companies.Cross sectional study.Professional associations for the 10 costliest disease areas in the US according to the US Agency for Healthcare Research and Quality. Financial data for association leadership, 2017-19, were obtained from the Open Payments database.328 leaders, such as board members, of 10 professional medical associations: American College of Cardiology, Orthopaedic Trauma Association, American Psychiatric Association, Endocrine Society, American College of Rheumatology, American Society of Clinical Oncology, American Thoracic Society, North American Spine Society, Infectious Diseases Society of America, and American College of Physicians.Proportion ...</code> | | <code>query: Récente recherche du Mexique sur l'ivermectine associée aux #azithromycine, #montelukast & aspirine sur 768 patients. La mortalité était diminuée de 81%, et 74% de baisse des hospitalisations. Un rétablissement 3,4 fois plus rapide pour les #Covid19. En prépublication</code> | <code>passage: title: Effectiveness of a multidrug therapy consisting of Ivermectin, Azithromycin, Montelukast, and Acetylsalicylic acid to prevent hospitalization and death among ambulatory COVID-19 cases in Tlaxcala, Mexico<br>abstract: There is an urgent need for effective treatments to prevent or attenuate lung and systemic inflammation, endotheliitis, and thrombosis related to COVID-19. This study aimed to assess the effectiveness of a multidrug-therapy consisting of Ivermectin, Azithromycin, Montelukast, and Acetylsalicylic acid ("TNR4" therapy) to prevent hospitalization and death among ambulatory COVID-19 cases in Tlaxcala, Mexico.</code> | | <code>query: 🧠💥 Suite à un #AVC, on peut constater une réponse inflammatoire persistante qui favorise alors une altération cognitive. La mise d'une molécule à des #souris 🐁 a autorisé de restaurer le métabolisme lipidique dans le #cerveau et diminuer ce risque</code> | <code>passage: in the striatum and thalamus and c-Fos immunoreactivity in hippocampal regions.

title: Repeated Administration of 2-Hydroxypropyl-β-Cyclodextrin (HPβCD) Attenuates the Chronic Inflammatory Response to Experimental Stroke Additionally, HPβCD improved recovery through the protection of hippocampal-dependent spatial working memory and reduction of impulsivity. These results indicate that systemic HPβCD treatment following stroke attenuates chronic inflammation and secondary neurodegeneration and prevents poststroke cognitive decline. SIGNIFICANCE STATEMENT Dementia is a common and debilitating sequela of stroke. Currently, there are no available treatments for poststroke dementia. Our study shows that lipid metabolism is disrupted in chronic stroke infarcts, which causes an accumulation of uncleared lipid debris and correlates with a chronic inflammatory response. To our knowledge, these substantial changes in lipid homeostasis have not been previously recognized or investigated...</code> |

• —Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }

Training Hyperparameters

Non-Default Hyperparameters

• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 32
• —num_train_epochs: 10
• —fp16: True
• —multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —prediction_loss_only: True
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 32
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1
• —num_train_epochs: 10
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.0
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: round_robin
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Training Time

• —Training: 1.1 hours

Framework Versions

• —Python: 3.12.6
• —Sentence Transformers: 5.4.1
• —Transformers: 4.56.0
• —PyTorch: 2.8.0+cu129
• —Accelerate: 1.10.1
• —Datasets: 4.8.4
• —Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->