Modelpublic

LocalWisdom/PurpleStatic

sourceHugging Facewtfplupdated 2mo agoView on Hugging Face

0likes

Model Card

PurpleStatic: Static Embeddings

This is a sentence-transformers model trained on the BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1, Heliosoph/Quora-Question-Pairs, owenkaplinsky/wildchat-paraphrases, mjbommar/ogbert-v1-contrastive and mjbommar/opengloss-v1.3-contrastive-examples datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

—Model Type: Sentence Transformer
—Maximum Sequence Length: inf tokens
—Output Dimensionality: 1024 dimensions
—Similarity Function: Cosine Similarity
—Supported Modality: Text
—Training Datasets:
—BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1
—Heliosoph/Quora-Question-Pairs
—owenkaplinsky/wildchat-paraphrases
—mjbommar/ogbert-v1-contrastive
—mjbommar/opengloss-v1.3-contrastive-examples
—Language: en
—License: wtfpl

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): StaticEmbedding({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("LocalWisdom/PurpleStatic")
# Run inference
sentences = [
    'How do you write a song?',
    'How do I write a song?',
    'How did Portugal become an independent country from Spain?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8566, 0.1409],
#         [0.8566, 1.0000, 0.1259],
#         [0.1409, 0.1259, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Datasets

<details><summary>BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1</summary>

BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

—Dataset: BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1
—Size: 5,694 training samples
—Columns: <code>anchor</code> and <code>positive</code>
—Approximate statistics based on the first 100 samples: | | anchor | positive | |:---------|:-------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------| | type | string | string | | modality | text | text | | details | <ul><li>min: 47 characters</li><li>mean: 135.34 characters</li><li>max: 352 characters</li></ul> | <ul><li>min: 70 characters</li><li>mean: 146.17 characters</li><li>max: 322 characters</li></ul> |
—Samples: | anchor | positive | |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Of our total net revenue of $10.8 billion in the fiscal year ended October 31, 2000, we generated 44.2% in the United States and 55.8% internationally.</code> | <code>In the fiscal year concluding October 31, 2000, the company produced 44.2% of its $10.8 billion total net revenue within the United States and 55.8% from international markets.</code> | | <code>Our sales strategy is to sell to and service our largest accounts (hospital and corporate business) directly while employing third-party distributors and manufacturer's representatives for smaller or more geographically dispersed countries.</code> | <code>We utilize a sales approach where we directly manage and sell to our major hospital and corporate clients, while relying on external distributors and manufacturer agents to reach smaller markets or those in remote locations.</code> | | <code>In August 2000, we announced a restructuring of our healthcare solutions business.</code> | <code>The company declared a reorganization of its healthcare solutions division in August 2000.</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

</details> <details><summary>Heliosoph/Quora-Question-Pairs</summary>

Heliosoph/Quora-Question-Pairs

—Dataset: Heliosoph/Quora-Question-Pairs
—Size: 149,263 training samples
—Columns: <code>anchor</code> and <code>positive</code>
—Approximate statistics based on the first 100 samples: | | anchor | positive | |:---------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------| | type | string | string | | modality | text | text | | details | <ul><li>min: 16 characters</li><li>mean: 54.75 characters</li><li>max: 139 characters</li></ul> | <ul><li>min: 21 characters</li><li>mean: 54.34 characters</li><li>max: 127 characters</li></ul> |
—Samples: | anchor | positive | |:----------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| | <code>Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?</code> | <code>I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?</code> | | <code>How can I be a good geologist?</code> | <code>What should I do to be a great geologist?</code> | | <code>How do I read and find my YouTube comments?</code> | <code>How can I see all my Youtube comments?</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

</details> <details><summary>owenkaplinsky/wildchat-paraphrases</summary>

owenkaplinsky/wildchat-paraphrases

—Dataset: owenkaplinsky/wildchat-paraphrases
—Size: 1,383,750 training samples
—Columns: <code>anchor</code> and <code>positive</code>
—Approximate statistics based on the first 100 samples: | | anchor | positive | |:---------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------| | type | string | string | | modality | text | text | | details | <ul><li>min: 20 characters</li><li>mean: 133.6 characters</li><li>max: 733 characters</li></ul> | <ul><li>min: 23 characters</li><li>mean: 129.4 characters</li><li>max: 639 characters</li></ul> |
—Samples: | anchor | positive | |:-------------------------------------------------------|:-------------------------------------------------------------------------| | <code>name current finance minister of pakistan</code> | <code>identify the present finance minister of pakistan</code> | | <code>name current finance minister of pakistan</code> | <code>who is pakistan's finance minister right now</code> | | <code>name current finance minister of pakistan</code> | <code>state the name of the current pakistani minister of finance</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

</details> <details><summary>mjbommar/ogbert-v1-contrastive</summary>

mjbommar/ogbert-v1-contrastive

—Dataset: mjbommar/ogbert-v1-contrastive
—Size: 2,673,774 training samples
—Columns: <code>anchor</code> and <code>positives</code>
—Approximate statistics based on the first 100 samples: | | anchor | positives | |:---------|:-----------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------| | type | string | string | | modality | text | text | | details | <ul><li>min: 4 characters</li><li>mean: 50.25 characters</li><li>max: 178 characters</li></ul> | <ul><li>min: 4 characters</li><li>mean: 36.68 characters</li><li>max: 121 characters</li></ul> |
—Samples: | anchor | positives | |:-------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | <code>The arrangement of halftone dots or grid that encodes tonal values in a halftone image.</code> | <code>The halftone dot grid encodes tonal levels.</code> | | <code>A formal subset of managers designated to handle day-to-day operations and implement board decisions.</code> | <code>The executive team meets monthly to review progress and adjust plans.</code> | | <code>niche market segment</code> | <code>submarket</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

</details> <details><summary>mjbommar/opengloss-v1.3-contrastive-examples</summary>

mjbommar/opengloss-v1.3-contrastive-examples

—Dataset: mjbommar/opengloss-v1.3-contrastive-examples
—Size: 161,496 training samples
—Columns: <code>anchor</code>, <code>positives</code>, <code>negatives1</code>, and <code>negatives2</code>
—Approximate statistics based on the first 100 samples: | | anchor | positives | negatives1 | negatives2 | |:---------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------| | type | string | string | string | string | | modality | text | text | text | text | | details | <ul><li>min: 37 characters</li><li>mean: 70.41 characters</li><li>max: 145 characters</li></ul> | <ul><li>min: 36 characters</li><li>mean: 71.77 characters</li><li>max: 146 characters</li></ul> | <ul><li>min: 33 characters</li><li>mean: 72.34 characters</li><li>max: 145 characters</li></ul> | <ul><li>min: 35 characters</li><li>mean: 71.88 characters</li><li>max: 146 characters</li></ul> |
—Samples: | anchor | positives | negatives1 | negatives2 | |:---------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------| | <code>Field interviews revealed a stony demeanor even when pressed for clarification.</code> | <code>Field interviews revealed a stoic demeanor even when pressed for clarification.</code> | <code>Field interviews revealed a warm demeanor even when pressed for clarification.</code> | <code>Field interviews revealed an empathetic demeanor even when pressed for clarification.</code> | | <code>The board followed formal protocol during the review.</code> | <code>The board followed official protocol during the review.</code> | <code>The board followed casual protocol during the review.</code> | <code>The board followed informal protocol during the review.</code> | | <code>The athlete carefully inspected the luge before the final run.</code> | <code>The athlete carefully inspected the toboggan before the final run.</code> | <code>The athlete carefully inspected the sled before the final run.</code> | <code>The athlete carefully inspected the bobsled before the final run.</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

</details>

Training Hyperparameters

Non-Default Hyperparameters

—per_device_train_batch_size: 4096
—num_train_epochs: 20
—learning_rate: 0.0001
—warmup_steps: 0.1
—bf16: True
—batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

—per_device_train_batch_size: 4096
—num_train_epochs: 20
—max_steps: -1
—learning_rate: 0.0001
—lr_scheduler_type: linear
—lr_scheduler_kwargs: None
—warmup_steps: 0.1
—optim: adamwtorchfused
—optim_args: None
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—optim_target_modules: None
—gradient_accumulation_steps: 1
—average_tokens_across_devices: True
—max_grad_norm: 1.0
—label_smoothing_factor: 0.0
—bf16: True
—fp16: False
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—use_liger_kernel: False
—liger_kernel_config: None
—use_cache: False
—neftune_noise_alpha: None
—torch_empty_cache_steps: None
—auto_find_batch_size: False
—log_on_each_node: True
—logging_nan_inf_filter: True
—include_num_input_tokens_seen: no
—log_level: passive
—log_level_replica: warning
—disable_tqdm: False
—project: huggingface
—trackio_space_id: None
—trackio_bucket_id: None
—trackio_static_space_id: None
—per_device_eval_batch_size: 8
—prediction_loss_only: True
—eval_on_start: False
—eval_do_concat_batches: True
—eval_use_gather_object: False
—eval_accumulation_steps: None
—include_for_metrics: []
—batch_eval_metrics: False
—save_only_model: False
—save_on_each_node: False
—enable_jit_checkpoint: False
—push_to_hub: False
—hub_private_repo: None
—hub_model_id: None
—hub_strategy: every_save
—hub_always_push: False
—hub_revision: None
—load_best_model_at_end: False
—ignore_data_skip: False
—restore_callback_states_from_checkpoint: False
—full_determinism: False
—seed: 42
—data_seed: None
—use_cpu: False
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—dataloader_prefetch_factor: None
—remove_unused_columns: True
—label_names: None
—train_sampling_strategy: random
—length_column_name: length
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—ddp_static_graph: None
—ddp_backend: None
—ddp_timeout: 1800
—fsdp: None
—fsdp_config: None
—deepspeed: None
—debug: []
—skip_memory_metrics: True
—do_predict: False
—resume_from_checkpoint: None
—warmup_ratio: None
—local_rank: -1
—prompts: None
—batch_sampler: no_duplicates
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss
0.0009	1	61.4605
0.9346	1000	45.8318
0.0009	1	60.6387
0.9346	1000	45.0978
1.8692	2000	44.8278
2.8037	3000	43.0278
3.7383	4000	41.6984
0.0009	1	55.3998
4.6729	5000	40.7913
5.6075	6000	40.1543
6.5421	7000	38.1761
7.4766	8000	38.8003
8.4112	9000	38.2068
9.3458	10000	37.4088
10.2804	11000	36.5805
11.2150	12000	37.1526
12.1495	13000	36.0028
13.0841	14000	35.2225
14.0187	15000	36.2388
14.9533	16000	35.1998
15.8879	17000	35.2619
16.8224	18000	35.1943
17.7570	19000	36.0396
18.6916	20000	34.2574
19.6262	21000	35.1139

Training Time

—Training: 2.1 hours

Framework Versions

—Python: 3.13.11
—Sentence Transformers: 5.6.1
—Transformers: 5.14.1
—PyTorch: 2.13.0+cu130
—Accelerate: 1.14.0
—Datasets: 5.0.0
—Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

bibtex

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

bibtex

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->