kaamd/ll

SentenceTransformer

This is a sentence-transformers model trained on the fiqa dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer <!-- - Base model: Unknown -->
• —Maximum Sequence Length: 32768 tokens
• —Output Dimensionality: 4096 dimensions
• —Similarity Function: Cosine Similarity
• —Training Dataset:
• —fiqa <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'MistralModel'})
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): SparseEmbedding(
    (sparsifyer): ZeroNeuron(in_features=4096, out_features=4096)
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: I am under 18 years old, in the US, my parents have terrible credit, how can I take out a loan?',
    "Depending on the state this might not be possible. Loans are considered contracts, and various states regulate how minors may enter into them. For example, in the state of Oregon, a minor may NOT enter into a contract without their parent being on the contract as well. So you are forced to wait until you turn 18. At that time you won't have a credit history, and to lenders that often is worse than having bad credit. I can't help with the car (other than to recommend you buy a junker for $500-$1,000 and just live with it for now), but you could certainly get a secured credit card or line of credit from your local bank. The way they are arranged is, you make a deposit of an amount of your choosing (generally at least $200 for credit cards, and $1,000 for lines of credit), and receive a revolving line with a limit of that same amount. As you use and pay on this loan, it will be reported in your credit history. If you start that now, by the time you turn 18 you will have much better options for purchasing vehicles.",
    "In the equity markets, the P/E is usually somewhere around 15.  The P/E can be viewed as the inverse of the rate of a perpetuity. Since the average is 15, and the E/P of that would be 6.7%, r should be 6.7% on average. If your business is growing, the growth rate can be incorporated like so: As you can see, a high g would make the price negative, in essence the seller should actually pay someone to take the business, but in reality, r is determined from the p and an estimated g. For a business of any growth rate, it's best to compare the multiple to the market, so for the average business in the market with your business's growth rate and industry, that P/E would be best applied to your company's income.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6277, 0.2807],
#         [0.6277, 1.0000, 0.2775],
#         [0.2807, 0.2775, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

fiqa

• —Dataset: fiqa
• —Size: 14,166 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 24 tokens</li><li>mean: 36.38 tokens</li><li>max: 62 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 223.61 tokens</li><li>max: 1683 tokens</li></ul> |
• —Samples: | anchor | positive | |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: What is considered a business expense on a business trip?</code> | <code>The IRS Guidance pertaining to the subject. In general the best I can say is your business expense may be deductible. But it depends on the circumstances and what it is you want to deduct. Travel Taxpayers who travel away from home on business may deduct related expenses, including the cost of reaching their destination, the cost of lodging and meals and other ordinary and necessary expenses. Taxpayers are considered “traveling away from home” if their duties require them to be away from home substantially longer than an ordinary day’s work and they need to sleep or rest to meet the demands of their work. The actual cost of meals and incidental expenses may be deducted or the taxpayer may use a standard meal allowance and reduced record keeping requirements. Regardless of the method used, meal deductions are generally limited to 50 percent as stated earlier. Only actual costs for lodging may be claimed as an expense and receipts must be kept for documentation. ...</code> | | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: Business Expense - Car Insurance Deductible For Accident That Occurred During a Business Trip</code> | <code>As a general rule, you must choose between a mileage deduction or an actual expenses deduction. The idea is that the mileage deduction is supposed to cover all costs of using the car. Exceptions include parking fees and tolls, which can be deducted separately under either method. You explicitly cannot deduct insurance costs if you claim a mileage deduction. Separately, you probably won't be able to deduct the deductible for your car as a casualty loss. You first subtract $100 from the deductible and then divide it by your Adjusted Gross Income (AGI) from your tax return. If your deductible is over 10% of your AGI, you can deduct it. Note that even with a $1500 deductible, you won't be able to deduct anything if you made more than $14,000 for the year. For most people, the insurance deductible just isn't large enough relative to income to be tax deductible. Source</code> | | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: Starting a new online business</code> | <code>Most US states have rules that go something like this: You will almost certainly have to pay some registration fees, as noted above. Depending on how you organize, you may or may not need to file a separate tax return for the business. (If you're sole proprietor for tax purposes, then you file on Schedule C on your personal Form 1040.) Whether or not you pay taxes depends on whether you have net income. It's possible that some losses might also be deductible. (Note that you may have to file a return even if you don't have net income - Filing and needing to pay are not the same since your return may indicate no tax due.) In addition, at the state level, you may have to pay additional fees or taxes beyond income tax depending on what you sell and how you sell it. (Sales tax, for example, might come into play as might franchise taxes.) You'll need to check your own state law for that. As always, it could be wise to get professional tax and accounting advice that's tailored to your si...</code> |
• —Loss: <code>zero-neuron.src.embedding.sparse_loss.SparseLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

• —per_device_eval_batch_size: 16
• —learning_rate: 0.0005
• —num_train_epochs: 2
• —lr_scheduler_type: cosine
• —warmup_ratio: 0.01
• —save_safetensors: False
• —bf16: True
• —remove_unused_columns: False
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: no
• —prediction_loss_only: True
• —per_device_train_batch_size: 8
• —per_device_eval_batch_size: 16
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 0.0005
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 2
• —max_steps: -1
• —lr_scheduler_type: cosine
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.01
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: True
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: False
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>

</details>

Framework Versions

• —Python: 3.10.16
• —Sentence Transformers: 5.1.1
• —Transformers: 4.55.4
• —PyTorch: 2.6.0+cu124
• —Accelerate: 1.8.0.dev0
• —Datasets: 3.2.0
• —Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->