vivnatan/snowflake-arctic-embed-l-medicare

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: Snowflake/snowflake-arctic-embed-l <!-- at revision d8fb21ca8d905d2832ee8b96c894d3298964346b -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vivnatan/snowflake-arctic-embed-l-medicare")
# Run inference
sentences = [
    'What are the consequences of not signing up for Medicare Part B when you are first eligible?',
    'Will I have to sign up for Part A and/or Part B?\nIf you’re close to 65, but NOT getting Social Security or RRB benefits, you’ll \nneed to sign up for Medicare. Visit SSA.gov/medicare to apply for Part A and \nPart B. You can also contact Social Security 3 months before you turn 65 to \nset up an appointment. If you worked for a railroad, visit RRB.gov, or call the \nRRB at 1-877-772-5772. TTY users can call 1-312-751-4701. \nIn most cases, if you don’t sign up for Part B when you’re first eligible, you may \nhave a delay in getting Medicare Part B coverage in the future because you \ncan only sign up at certain times. You may also have to pay a late enrollment \npenalty for as long as you have Part B. Go to page 23.',
    '•\t Visit ACL.gov/ltc to learn more about planning for long‑term care. \n•\t Visit the Eldercare Locator at eldercare.acl.gov, or call 1-800-677-1116 to \nfind help in your community.\n•\t Call your Long-Term Care Ombudsman, or visit ltcombudsman.org for help \nwith services you need and to be advised of your rights, and to find an \nOmbudsman program near you.\n•\t Call your State Medical Assistance (Medicaid) office or visit Medicaid.gov \nand ask for information about long-term care coverage. \n•\t Call your State Health Insurance Assistance Program (SHIP). Go to \npages 114–117 for the phone number of your local SHIP. \n•\t Call your State Insurance Department for information on long‑term care \ninsurance. Call 1-800-MEDICARE (1‑800‑633‑4227) to get the phone \nnumber. TTY users can call 1‑877‑486‑2048.\n•\t Get a copy of “A Shopper’s Guide to Long-Term Care Insurance” from the \nNational Association of Insurance Commissioners at content.naic.org/sites/\ndefault/files/publication-ltc-lp-shoppers-guide-long-term.pdf.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 910 training samples
• —Columns: <code>sentence0</code> and <code>sentence1</code>
• —Approximate statistics based on the first 910 samples: | | sentence0 | sentence1 | |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 10 tokens</li><li>mean: 19.78 tokens</li><li>max: 44 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 312.08 tokens</li><li>max: 512 tokens</li></ul> |
• —Samples: | sentence0 | sentence1 | |:----------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>What benefits do Medicare Advantage Plans provide compared to Original Medicare?</code> | <code>62<br>Section 4: Medicare Advantage Plans & other options<br>62<br>What do Medicare Advantage Plans cover? <br>Medicare Advantage Plans provide almost all of your Part A and Part B <br>benefits, including most new benefits that come from laws or Medicare <br>policy decisions. Medicare Advantage Plan benefits exclude hospice care <br>and some costs of clinical trials. But if you’re in a Medicare Advantage Plan, <br>Original Medicare will still help cover your costs for hospice care and some <br>costs for clinical research studies, and benefits that come from laws or <br>Medicare policy decisions that the plan doesn't cover. The plan can choose not <br>to cover the costs of services that aren’t medically necessary under Medicare. <br>In some instances, where Medicare hasn’t established coverage criteria, plans <br>may also use their own coverage criteria to determine if certain services are <br>medically necessary. If you aren’t sure whether a service is covered, check <br>with your provider before you get the service. If you disag...</code> | | <code>Which services are excluded from coverage under Medicare Advantage Plans?</code> | <code>62<br>Section 4: Medicare Advantage Plans & other options<br>62<br>What do Medicare Advantage Plans cover? <br>Medicare Advantage Plans provide almost all of your Part A and Part B <br>benefits, including most new benefits that come from laws or Medicare <br>policy decisions. Medicare Advantage Plan benefits exclude hospice care <br>and some costs of clinical trials. But if you’re in a Medicare Advantage Plan, <br>Original Medicare will still help cover your costs for hospice care and some <br>costs for clinical research studies, and benefits that come from laws or <br>Medicare policy decisions that the plan doesn't cover. The plan can choose not <br>to cover the costs of services that aren’t medically necessary under Medicare. <br>In some instances, where Medicare hasn’t established coverage criteria, plans <br>may also use their own coverage criteria to determine if certain services are <br>medically necessary. If you aren’t sure whether a service is covered, check <br>with your provider before you get the service. If you disag...</code> | | <code>What should you do if you are unsure whether a service is covered by your Medicare Advantage Plan?</code> | <code>62<br>Section 4: Medicare Advantage Plans & other options<br>62<br>What do Medicare Advantage Plans cover? <br>Medicare Advantage Plans provide almost all of your Part A and Part B <br>benefits, including most new benefits that come from laws or Medicare <br>policy decisions. Medicare Advantage Plan benefits exclude hospice care <br>and some costs of clinical trials. But if you’re in a Medicare Advantage Plan, <br>Original Medicare will still help cover your costs for hospice care and some <br>costs for clinical research studies, and benefits that come from laws or <br>Medicare policy decisions that the plan doesn't cover. The plan can choose not <br>to cover the costs of services that aren’t medically necessary under Medicare. <br>In some instances, where Medicare hasn’t established coverage criteria, plans <br>may also use their own coverage criteria to determine if certain services are <br>medically necessary. If you aren’t sure whether a service is covered, check <br>with your provider before you get the service. If you disag...</code> |
• —Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 10
• —per_device_eval_batch_size: 10
• —num_train_epochs: 10
• —multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 10
• —per_device_eval_batch_size: 10
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1
• —num_train_epochs: 10
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.0
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —tp_size: 0
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: round_robin

</details>

Training Logs

Framework Versions

• —Python: 3.11.12
• —Sentence Transformers: 4.1.0
• —Transformers: 4.51.3
• —PyTorch: 2.6.0+cu124
• —Accelerate: 1.6.0
• —Datasets: 3.6.0
• —Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->