FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_32

SentenceTransformer based on flax-sentence-embeddings/alldatasetsv4_MiniLM-L6

This is a sentence-transformers model finetuned from flax-sentence-embeddings/all_datasets_v4_MiniLM-L6 on the json dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: flax-sentence-embeddings/all_datasets_v4_MiniLM-L6 <!-- at revision a407cc0b7d85eec9a5617eaf51dbe7b353b0c79f -->
• —Maximum Sequence Length: 128 tokens
• —Output Dimensionality: 384 tokens
• —Similarity Function: Cosine Similarity
• —Training Dataset:
• —json <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_32")
# Run inference
sentences = [
    '\nAtypical hemolytic uremic syndrome (aHUS) with H factor anomaly is a disease characterized by an atypical form of hemolytic uremic syndrome, a severe thrombotic microangiopathy that leads to kidney failure, anemia, and thrombocytopenia. This specific subtype of aHUS is notable for its association with an anomaly in the H factor, potentially involving complement system dysregulation. As such, it falls under the broader category of hemolytic uremic syndrome, a condition marked by differential diagnosis complexity and distinct etiologies. Patients with aHUS often require a nuanced approach to diagnosis and management, emphasizing awareness of its distinct characteristics in comparison with other forms of hemolytic uremic syndrome, ensuring comprehensive and accurate differential diagnosis which might include conditions like thrombotic thrombocytopenic purpura (TTP) or disseminated intravascular coagulation (DIC). The identification and management of aHUS with H factor anomaly necessitates multidisciplinary collaboration and up-to-date knowledge alongside genetic and clinical features specific to this condition.',
    'Could you list the diseases related to or subtypes of type 1 atypical hemolytic uremic syndrome for differential diagnosis purposes?',
    'Which diseases are associated with anomalies in the CD4 gene or protein, alongside genetic mutations that impact muscle protein synthesis?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: dim_384
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

json

• —Dataset: json
• —Size: 1,814 training samples
• —Columns: <code>positive</code> and <code>anchor</code>
• —Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 3 tokens</li><li>mean: 118.96 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 34.93 tokens</li><li>max: 116 tokens</li></ul> |
• —Samples: | positive | anchor | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code><br><br>Epilepsy is a neurological disorder characterized by recurrent seizures, which can be sudden, abnormal electrical events in the brain. Seizures can affect different parts of the brain and range from mild to severe. Symptoms can include muscle stiffness, twitching, loss of consciousness, and cognitive disruptions. Seizures can be divided into focal (partial) seizures and generalized seizures.<br><br>### Causes:<br>1. Brain Injury or Infection: These can lead to scar tissue and abnormal electrical activity.<br>2. Developmental Abnormalities: Genetic or developmental issues can cause seizures.<br>3. Brain Tumors or Bleeds: These can result in seizures.<br>4. Inherited Disorders: Some genetic conditions lead to epilepsy.<br><br>### Complications:<br>- Status Epilepticus: Continuous seizure activity lasting more than five minutes.<br>- Sudden Unexpected Death in Epilepsy (SUDEP): Unexplained death during an untreated condition, especially if seizures aren't controlled.<br>- Emotional Health Issues: Increased risk for depression, anxiety, and</code> | <code>Search for medical conditions not treatable by any known medications that present with hoarseness as a symptom.</code> | | <code><br>Diphyllobothriasis, also known as bothriocephalosis, is a parasitosis caused by the intestinal infection with the larval stage of the tapeworm Diphyllobothrium. This condition is characterized by a broad array of symptoms, including frequent stomach discomfort, nausea, appetite loss, fatigue, and weakness. These symptoms are medically attributed to anemia, which stems from vitamin B12 deficiency—a common complication linked to this parasitosis. The anemia caused by diphyllobothriasis can also resemble Biermer's anemia, distinguished by abnormally large red blood cells. Individuals with a family history of ceestode infections, such as diphyllobothriasis, and those who exhibit symptoms such as those described, may be more susceptible to this condition. The disease, which is cosmopolitan in nature, has been reported in Europe, primarily in areas like the Italian, Swiss, and French Alps, though its prevalence across the continent remains unknown. Treatment for diphyllobothriasis typically involves the use of standard medications such as niclosamide or praziquantel, which are effective in clearing the parasite.</code> | <code>What could be the condition causing frequent stomach discomfort, nausea, appetite loss, fatigue, and weakness in me, possibly linked to a family history of Cestode infection and associated with vitamin B12 deficiency and abnormal red blood cells resembling Biermer's anemia symptoms?</code> | | <code><br>The provided list appears to be a collection of gene names. Genes are segments of DNA that code for proteins and play a crucial role in various biological functions, influencing traits, growth, and processes within an organism. They are fundamental units of heredity. The presence of these gene names suggests that the document is most likely related to genetic research, medical studies, or bioinformatics. This could involve analyses of genetic sequences, expression patterns, or functional assays related to the specific genes mentioned, possibly with the aim of understanding genetic disorders, development, or disease mechanisms.</code> | <code>Which cellular structures engage in interactions with genes or proteins that are affected by the administration of Mevastatin?</code> |
• —Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: epoch
• —per_device_train_batch_size: 32
• —learning_rate: 1e-05
• —num_train_epochs: 10
• —warmup_ratio: 0.1
• —bf16: True
• —tf32: False
• —load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: epoch
• —prediction_loss_only: True
• —per_device_train_batch_size: 32
• —per_device_eval_batch_size: 8
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 1e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 10
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: True
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: False
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: False
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

• —The bold row denotes the saved checkpoint.

Framework Versions

• —Python: 3.10.10
• —Sentence Transformers: 3.1.1
• —Transformers: 4.45.1
• —PyTorch: 2.2.1+cu121
• —Accelerate: 0.34.2
• —Datasets: 3.0.1
• —Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->