capemox/splade-ettin-encoder-17m-gooaq

splade-ettin-encoder-17m trained on GooAQ

This is a SPLADE Sparse Encoder model finetuned from jhu-clsp/ettin-encoder-17m on the gooaq dataset using the sentence-transformers library. It maps sentences & paragraphs to a 50368-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• —Model Type: SPLADE Sparse Encoder
• —Base model: jhu-clsp/ettin-encoder-17m <!-- at revision 987607455c61e7a5bbc85f7758e0512ea6d0ae4c -->
• —Maximum Sequence Length: 256 tokens
• —Output Dimensionality: 50368 dimensions
• —Similarity Function: Dot Product
• —Supported Modality: Text
• —Training Dataset:
• —gooaq
• —Language: en
• —License: mit

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Documentation: Sparse Encoder Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 50368})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("capemox/splade-ettin-encoder-17m-gooaq")
# Run inference
sentences = [
    'how close is the closest star?',
    'The two main stars are Alpha Centauri A and Alpha Centauri B, which form a binary pair. They are an average of 4.3 light-years from Earth. The third star is Proxima Centauri. It is about 4.22 light-years from Earth and is the closest star other than the sun.',
    'One gallon can of paint will cover up to 400 square feet, which is enough to cover a small room like a bathroom. Two gallon cans of paint cover up to 800 square feet, which is enough to cover an average size room. This is the most common amount needed, especially when considering second coat coverage.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 50368]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[144.9772, 158.3885, 140.6785],
#         [158.3885, 496.0056, 344.2902],
#         [140.6785, 344.2902, 528.3322]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Sparse Information Retrieval

• —Datasets: NanoMSMARCO_256, NanoNFCorpus_256, NanoNQ_256, NanoClimateFEVER_256, NanoDBPedia_256, NanoFEVER_256, NanoFiQA2018_256, NanoHotpotQA_256, NanoMSMARCO_256, NanoNFCorpus_256, NanoNQ_256, NanoQuoraRetrieval_256, NanoSCIDOCS_256, NanoArguAna_256, NanoSciFact_256 and NanoTouche2020_256
• —Evaluated with <code>SparseInformationRetrievalEvaluator</code> with these parameters:

  {
      "max_active_dims": 256
  }

Sparse Nano BEIR

• —Dataset: NanoBEIR_mean_256
• —Evaluated with <code>SparseNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "max_active_dims": 256
  }

Sparse Nano BEIR

• —Dataset: NanoBEIR_mean_256
• —Evaluated with <code>SparseNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "max_active_dims": 256
  }

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

gooaq

• —Dataset: gooaq at b089f72
• —Size: 99,000 training samples
• —Columns: <code>question</code> and <code>answer</code>
• —Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 8 tokens</li><li>mean: 11.93 tokens</li><li>max: 21 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 58.27 tokens</li><li>max: 137 tokens</li></ul> |
• —Samples: | question | answer | |:-----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>what are the 5 characteristics of a star?</code> | <code>Key Concept: Characteristics used to classify stars include color, temperature, size, composition, and brightness.</code> | | <code>are copic markers alcohol ink?</code> | <code>Copic Ink is alcohol-based and flammable. Keep away from direct sunlight and extreme temperatures.</code> | | <code>what is the difference between appellate term and appellate division?</code> | <code>Appellate terms An appellate term is an intermediate appellate court that hears appeals from the inferior courts within their designated counties or judicial districts, and are intended to ease the workload on the Appellate Division and provide a less expensive forum closer to the people.</code> |
• —Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
      "document_regularizer_weight": 3e-05,
      "query_regularizer_weight": 5e-05
  }

Evaluation Dataset

gooaq

• —Dataset: gooaq at b089f72
• —Size: 1,000 evaluation samples
• —Columns: <code>question</code> and <code>answer</code>
• —Approximate statistics based on the first 1000 samples: | | question | answer | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 8 tokens</li><li>mean: 12.05 tokens</li><li>max: 23 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 58.98 tokens</li><li>max: 186 tokens</li></ul> |
• —Samples: | question | answer | |:-----------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>should you take ibuprofen with high blood pressure?</code> | <code>In general, people with high blood pressure should use acetaminophen or possibly aspirin for over-the-counter pain relief. Unless your health care provider has said it's OK, you should not use ibuprofen, ketoprofen, or naproxen sodium. If aspirin or acetaminophen doesn't help with your pain, call your doctor.</code> | | <code>how old do you have to be to work in sc?</code> | <code>The general minimum age of employment for South Carolina youth is 14, although the state allows younger children who are performers to work in show business. If their families are agricultural workers, children younger than age 14 may also participate in farm labor.</code> | | <code>how to write a topic proposal for a research paper?</code> | <code>['Write down the main topic of your paper. ... ', 'Write two or three short sentences under the main topic that explain why you chose that topic. ... ', 'Write a thesis sentence that states the angle and purpose of your research paper. ... ', 'List the items you will cover in the body of the paper that support your thesis statement.']</code> |
• —Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
      "document_regularizer_weight": 3e-05,
      "query_regularizer_weight": 5e-05
  }

Training Hyperparameters

Non-Default Hyperparameters

• —per_device_train_batch_size: 32
• —num_train_epochs: 1
• —learning_rate: 2e-05
• —bf16: True
• —per_device_eval_batch_size: 32
• —load_best_model_at_end: True
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —per_device_train_batch_size: 32
• —num_train_epochs: 1
• —max_steps: -1
• —learning_rate: 2e-05
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: None
• —warmup_steps: 0
• —optim: adamwtorchfused
• —optim_args: None
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —optim_target_modules: None
• —gradient_accumulation_steps: 1
• —average_tokens_across_devices: True
• —max_grad_norm: 1.0
• —label_smoothing_factor: 0.0
• —bf16: True
• —fp16: False
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —use_liger_kernel: False
• —liger_kernel_config: None
• —use_cache: False
• —neftune_noise_alpha: None
• —torch_empty_cache_steps: None
• —auto_find_batch_size: False
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —include_num_input_tokens_seen: no
• —log_level: passive
• —log_level_replica: warning
• —disable_tqdm: False
• —project: huggingface
• —trackio_space_id: trackio
• —per_device_eval_batch_size: 32
• —prediction_loss_only: True
• —eval_on_start: False
• —eval_do_concat_batches: True
• —eval_use_gather_object: False
• —eval_accumulation_steps: None
• —include_for_metrics: []
• —batch_eval_metrics: False
• —save_only_model: False
• —save_on_each_node: False
• —enable_jit_checkpoint: False
• —push_to_hub: False
• —hub_private_repo: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_always_push: False
• —hub_revision: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —restore_callback_states_from_checkpoint: False
• —full_determinism: False
• —seed: 42
• —data_seed: None
• —use_cpu: False
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —dataloader_prefetch_factor: None
• —remove_unused_columns: True
• —label_names: None
• —train_sampling_strategy: random
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —ddp_backend: None
• —ddp_timeout: 1800
• —fsdp: []
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —deepspeed: None
• —debug: []
• —skip_memory_metrics: True
• —do_predict: False
• —resume_from_checkpoint: None
• —warmup_ratio: None
• —local_rank: -1
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

• —The bold row denotes the saved checkpoint.

Training Time

• —Training: 7.7 minutes
• —Evaluation: 2.0 minutes
• —Total: 9.7 minutes

Framework Versions

• —Python: 3.12.3
• —Sentence Transformers: 5.4.1
• —Transformers: 5.5.4
• —PyTorch: 2.11.0+cu130
• —Accelerate: 1.13.0
• —Datasets: 4.8.4
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->