ayushexel/colbert-MiniLM-L6-H384-1-epoch-gooaq-1995000

ColBERT based on nreimers/MiniLM-L6-H384-uncased

This is a sentence-transformers model finetuned from nreimers/MiniLM-L6-H384-uncased. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: nreimers/MiniLM-L6-H384-uncased <!-- at revision 3276f0fac9d818781d7a1327b3ff818fc4e643c0 -->
• —Maximum Sequence Length: 31 tokens
• —Output Dimensionality: 128 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

ColBERT(
  (0): Transformer({'max_seq_length': 31, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Dense({'in_features': 384, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'are emg pickups any good?',
    "EMGs are a one trick pony, and only sound good for high gain applications. Sort of, they definitely aren't as flexible as most passive options, but most metal oriented passive pickups have the same issue. A lot of guitarists forget that EMG makes more pickups than just the 81/85 set.",
    "Among guitar and bass accessories, the company sells active humbucker pickups, such as the EMG 81, the EMG 85, the EMG 60, and the EMG 89. They also produce passive pickups such as the EMG-HZ Series, which include SRO-OC1's and SC Sets.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 1,893,949 training samples
• —Columns: <code>question</code>, <code>answer</code>, and <code>negative</code>
• —Approximate statistics based on the first 1000 samples: | | question | answer | negative | |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | string | | details | <ul><li>min: 9 tokens</li><li>mean: 13.01 tokens</li><li>max: 27 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 31.78 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 31.66 tokens</li><li>max: 32 tokens</li></ul> |
• —Samples: | question | answer | negative | |:-------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>what is the relationship between humility and thankfulness?</code> | <code>how gratitude can influence humility and vice versa. Humility is characterized by low self-focus, secure sense of self, and increased valuation of others. Gratitude is marked by a sense that one has benefited from the actions of another.</code> | <code>-hum-, root. -hum- comes from Latin, where it has the meaning "ground. '' This meaning is found in such words as: exhume, humble, humiliate, humility, humus, posthumous.</code> | | <code>what is the difference between usb a b c?</code> | <code>The USB-A has a much larger physical connector than the Type C, Type C is around the same size as a micro-USB connector. Unlike, Type A, you won't need to try and insert it, flip it over and then flip it over once more just to find the right orientation when trying to make a connection.</code> | <code>First the transfer rates: USB 2.0 offers transfer rates of 480 Mbps and USB 3.0 offers transfer rates of 4.8 Gbps - that's 10 times faster. ... USB 2.0 provided up to 500 mA whereas USB 3.0 provides up to 900 mA, allowing power hungry devices to now be bus powered.</code> | | <code>how hyaluronic acid is made?</code> | <code>Hyaluronic acid is a substance that is naturally present in the human body. It is found in the highest concentrations in fluids in the eyes and joints. The hyaluronic acid that is used as medicine is extracted from rooster combs or made by bacteria in the laboratory.</code> | <code>Hyaluronic acid helps your skin hang on to the moisture. 2. ... Hyaluronic acid by itself is non-comedogenic (doesn't clog pores), but you should be careful when choosing a hyaluronic acid serum that the ingredient list doesn't contain any sneaky pore-clogging ingredients you're not expecting.</code> |
• —Loss: <code>pylate.losses.contrastive.Contrastive</code>

Evaluation Dataset

Unnamed Dataset

• —Size: 5,000 evaluation samples
• —Columns: <code>question</code>, <code>answer</code>, and <code>negative_1</code>
• —Approximate statistics based on the first 1000 samples: | | question | answer | negative_1 | |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | string | | details | <ul><li>min: 9 tokens</li><li>mean: 12.96 tokens</li><li>max: 22 tokens</li></ul> | <ul><li>min: 19 tokens</li><li>mean: 31.7 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 31.43 tokens</li><li>max: 32 tokens</li></ul> |
• —Samples: | question | answer | negative_1 | |:----------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>are tefal ingenio pans suitable for induction hobs?</code> | <code>Tefal Ingenio is a revolutionary concept that brings a whole new take on versatility. ... The frying pans also feature Tefal's iconic Thermo-Spot which lets you know when the pan has reached optimal cooking temperature. The Ingenio Induction range is compatible with all hobs and is also dishwasher safe.</code> | <code>Tefal Ingenio is a revolutionary concept that brings a whole new take on versatility. ... The frying pans also feature Tefal's iconic Thermo-Spot which lets you know when the pan has reached optimal cooking temperature. The Ingenio Induction range is compatible with all hobs and is also dishwasher safe.</code> | | <code>how many continuing education hours is acls?</code> | <code>The ACLS, PALS, and NRP certification courses are approved for 8 CEUs/CMEs, and recertification courses are approved for 4 CEUs/CMEs. The BLS certification course is approved for 4 CEUs/CMEs and the recertification course is approved for 2 CEUs/CMEs. For more information, please visit our Accreditation page.</code> | <code>The foremost difference between the two is their advancement level. Essentially, ACLS is a sophisticated and more advanced course and builds upon the major fundamentals developed during BLS. The main purpose of BLS and ACLS certification are well explained in this article.</code> | | <code>what are the health benefits of drinking peppermint tea?</code> | <code>['Makes you Stress Free. When it comes to relieving stress and anxiety, peppermint tea is one of the best allies. ... ', 'Sleep-Friendly. ... ', 'Aids in Weight Loss. ... ', 'Cure for an Upset Stomach. ... ', 'Improves Digestion. ... ', 'Boosts Immune System. ... ', 'Fights Bad Breath.']</code> | <code>Peppermint tea is a popular herbal tea that is naturally calorie- and caffeine-free. Some research has suggested that the oils in peppermint may have a number of other health benefits, such as fresher breath, better digestion, and reduced pain from headaches. Peppermint tea also has antibacterial properties.</code> |
• —Loss: <code>pylate.losses.contrastive.Contrastive</code>

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 256
• —per_device_eval_batch_size: 256
• —learning_rate: 3e-06
• —num_train_epochs: 1
• —warmup_ratio: 0.1
• —seed: 12
• —bf16: True
• —dataloader_num_workers: 12
• —load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 256
• —per_device_eval_batch_size: 256
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 3e-06
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 1
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 12
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: True
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 12
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —tp_size: 0
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• —Python: 3.11.0
• —Sentence Transformers: 4.0.1
• —Transformers: 4.50.3
• —PyTorch: 2.6.0+cu124
• —Accelerate: 1.5.2
• —Datasets: 3.5.0
• —Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->