Modelpublic

CatkinChen/sentence-transformers_all-MiniLM-L6-v2_retrieval_finetuned_2025-03-29_23-45-57

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes84downloads

Model Card

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: sentence-transformers/all-MiniLM-L6-v2
—Maximum Sequence Length: 256 tokens
—Output Dimensionality: 384 dimensions
—Similarity Function: Cosine Similarity

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("CatkinChen/sentence-transformers_all-MiniLM-L6-v2_retrieval_finetuned_2025-03-29_23-45-57")
# Run inference
sentences = [
    'What is the name of the spell that removes light?',
    'Book: 3, Chapter: 17\nPassage: He raised his eyebrows at her; she nodded again and let go. Quietly as they could, they crept out into the hall and up the crumbling staircase. Everything was covered in a thick layer of dust except the floor, where a wide shiny stripe had been made by something being dragged upstairs. They reached the dark landing. "Nox," they whispered together, and the lights at the end of their wands went out. Only one door was open. As they crept toward it, they heard movement from behind it; a low moan, and then a deep, loud purring. They exchanged a last look, a last nod. Wand held tightly before him, Harry kicked the door wide open.',
    'Book: 5, Chapter: 15\nPassage: "Uh ... first year - you saved the Stone from You-Know-Who." "But that was luck," said Harry, "that wasn\'t skill -"\n"Second year," Ron interrupted, "you killed the basilisk and destroyed Riddle." "Yeah, but if Fawkes hadn\'t turned up I -"\n"Third year," said Ron, louder still, "you fought off about a hundred dementors at once -"\n"You know that was a fluke, if the Time-Turner hadn\'t -"\n"Last year," Ron said, almost shouting now, "you fought off You-Know-Who again -"\n"Listen to me!" said Harry, almost angrily, because Ron and Hermione were both smirking now. "Just listen to me, all right? It sounds great when you say it like that, but all that stuff was luck - I didn\'t know what I was doing half the time, I didn\'t plan any of it, I just did whatever I could think of, and I nearly always had help -"\nRon and Hermione were still smirking and Harry felt his temper rise; he wasn\'t even sure why he was feeling so angry. "Don\'t sit there grinning like you know better than I do, I was there, wasn\'t I?" he said heatedly. "I know what went on, all right? And I didn\'t get through any of that because I was brilliant at Defense Against the Dark Arts, I got through it all because - because help came at the right time, or because I guessed right - but I just blundered through it all, I didn\'t have a clue what I was doing - STOP LAUGHING!" The bowl of murtlap essence fell to the floor and smashed. He became aware that he was on his feet, though he couldn\'t remember standing up. Crookshanks streaked away under a sofa; Ron and Hermione\'s smiles had vanished. "You don\'t know what it\'s like!',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Triplet

—Dataset: test-eval
—Evaluated with <code>TripletEvaluator</code>

Metric	Value
cosine_accuracy	0.8857

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

—Size: 309 training samples
—Columns: <code>sentence0</code>, <code>sentence1</code>, and <code>sentence_2</code>
—Approximate statistics based on the first 309 samples: | | sentence0 | sentence1 | sentence_2 | |:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | string | | details | <ul><li>min: 9 tokens</li><li>mean: 17.74 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 46 tokens</li><li>mean: 187.05 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 37 tokens</li><li>mean: 165.52 tokens</li><li>max: 256 tokens</li></ul> |
—Samples: | sentence0 | sentence1 | sentence_2 | |:------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Who is the supposed criminal believed to be after Harry?</code> | <code>Book: 3, Chapter: 5 Passage: "He looks like one good hex would finish him off, doesn't he? Anyway ..." He turned to Harry. "What were you going to tell us?" Harry explained all about Mr. and Mrs. Weasley's argument and the warning Mr. Weasley had just given him. When he'd finished, Ron looked thunderstruck, and Hermione had her hands over her mouth. She finally lowered them to say, "Sirius Black escaped to come after you? Oh, Harry ... you'll have to be really, really careful. Don't go looking for trouble, Harry -" "I don't go looking for trouble," said Harry, nettled. "Trouble usually finds me." "How thick would Harry have to be, to go looking for a nutter who wants to kill him?"</code> | <code>Book: 6, Chapter: 9 Passage: "I do not take cheek from anyone, Potter ... not even 'the Chosen One.' " "That was brilliant, Harry!" chortled Ron, once they were safely on their way to break a short while later. "You really shouldn't have said it," said Hermione, frowning at Ron. "What made you?"</code> | | <code>In the sixth book, what does Harry see in the Pensieve about Voldemort's mother, and how does it connect to the second book?</code> | <code>Book: 6, Chapter: 10 Passage: "Voldemort's grandfather, yes," said Dumbledore. "Marvolo, his son, Morfin, and his daughter, Merope, were the last of the Gaunts, a very ancient Wizarding family noted for a vein of instability and violence that flourished through the generations due to their habit of marrying their own cousins. Lack of sense coupled with a great liking for grandeur meant that the family gold was squandered several generations before Marvolo was born. He, as you saw, was left in squalor and poverty, with a very nasty temper, a fantastic amount of arrogance and pride, and a couple of family heirlooms that he treasured just as much as his son, and rather more than his daughter." "So Merope," said Harry, leaning forward in his chair and staring at Dumbledore, "so Merope was ... Sir, does that mean she was ... Voldemort's mother?" "It does," said Dumbledore. "And it so happens that we also had a glimpse of Voldemort's father. I wonder whether you noticed?" "The Muggle Morfin ...</code> | <code>Book: 5, Chapter: 35 Passage: Why?" "Why?" Malfoy sounded incredulously delighted. "Because the only people who are permitted to retrieve a prophecy from the Department of Mysteries, Potter, are those about whom it was made, as the Dark Lord discovered when he attempted to use others to steal it for him." "And why did he want to steal a prophecy about me?" "About both of you, Potter, about both of you ... Haven't you ever wondered why the Dark Lord tried to kill you as a baby?"</code> | | <code>Who is the charms master who helps with the Triwizard Tournament?</code> | <code>Book: 7, Chapter: 30 Passage: "Professor," Harry said, approaching the little Charms master, "Professor, I'm sorry to interrupt, but this is important. Have you got any idea where the diadem of Ravenclaw is?" "- Protego Horribilis - the diadem of Ravenclaw?" squeaked Flitwick. "A little extra wisdom never goes amiss, Potter, but I hardly think it would be much use in this situation!" "I only meant - do you know where it is? Have you ever seen it?" "Seen it? Nobody has seen it in living memory! Long since lost, boy!" Harry felt a mixture of desperate disappointment and panic. What, then, was the Horcrux? "We shall meet you and your Ravenclaws in the Great Hall, Filius!"</code> | <code>Book: 5, Chapter: 6 Passage: "What do you want anyway?" George asked. Kreacher's huge eyes darted onto George. "Kreacher is cleaning," he said evasively. "A likely story," said a voice behind Harry.</code> |
—Loss: <code>TripletLoss</code> with these parameters:

json

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.5
  }

Training Hyperparameters

Non-Default Hyperparameters

—eval_strategy: steps
—per_device_train_batch_size: 16
—per_device_eval_batch_size: 16
—num_train_epochs: 10
—multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

—overwrite_output_dir: False
—do_predict: False
—eval_strategy: steps
—prediction_loss_only: True
—per_device_train_batch_size: 16
—per_device_eval_batch_size: 16
—per_gpu_train_batch_size: None
—per_gpu_eval_batch_size: None
—gradient_accumulation_steps: 1
—eval_accumulation_steps: None
—torch_empty_cache_steps: None
—learning_rate: 5e-05
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—max_grad_norm: 1
—num_train_epochs: 10
—max_steps: -1
—lr_scheduler_type: linear
—lr_scheduler_kwargs: {}
—warmup_ratio: 0.0
—warmup_steps: 0
—log_level: passive
—log_level_replica: warning
—log_on_each_node: True
—logging_nan_inf_filter: True
—save_safetensors: True
—save_on_each_node: False
—save_only_model: False
—restore_callback_states_from_checkpoint: False
—no_cuda: False
—use_cpu: False
—use_mps_device: False
—seed: 42
—data_seed: None
—jit_mode_eval: False
—use_ipex: False
—bf16: False
—fp16: False
—fp16_opt_level: O1
—half_precision_backend: auto
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—local_rank: 0
—ddp_backend: None
—tpu_num_cores: None
—tpu_metrics_debug: False
—debug: []
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_prefetch_factor: None
—past_index: -1
—disable_tqdm: False
—remove_unused_columns: True
—label_names: None
—load_best_model_at_end: False
—ignore_data_skip: False
—fsdp: []
—fsdp_min_num_params: 0
—fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
—tp_size: 0
—fsdp_transformer_layer_cls_to_wrap: None
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—deepspeed: None
—label_smoothing_factor: 0.0
—optim: adamw_torch
—optim_args: None
—adafactor: False
—group_by_length: False
—length_column_name: length
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—skip_memory_metrics: True
—use_legacy_prediction_loop: False
—push_to_hub: False
—resume_from_checkpoint: None
—hub_model_id: None
—hub_strategy: every_save
—hub_private_repo: None
—hub_always_push: False
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—include_inputs_for_metrics: False
—include_for_metrics: []
—eval_do_concat_batches: True
—fp16_backend: auto
—push_to_hub_model_id: None
—push_to_hub_organization: None
—mp_parameters:
—auto_find_batch_size: False
—full_determinism: False
—torchdynamo: None
—ray_scope: last
—ddp_timeout: 1800
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—dispatch_batches: None
—split_batches: None
—include_tokens_per_second: False
—include_num_input_tokens_seen: False
—neftune_noise_alpha: None
—optim_target_modules: None
—batch_eval_metrics: False
—eval_on_start: False
—use_liger_kernel: False
—eval_use_gather_object: False
—average_tokens_across_devices: False
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: round_robin

</details>

Training Logs

Epoch	Step	test-eval_cosine_accuracy
0.5	10	0.8000
1.0	20	0.8071
1.5	30	0.8500
2.0	40	0.8429
2.5	50	0.8714
3.0	60	0.9000
3.5	70	0.8929
4.0	80	0.8929
4.5	90	0.8929
5.0	100	0.9000
5.5	110	0.9000
6.0	120	0.9000
6.5	130	0.8929
7.0	140	0.8857
7.5	150	0.8929
8.0	160	0.8929
8.5	170	0.8857
9.0	180	0.8857
9.5	190	0.8857
10.0	200	0.8857
-1	-1	0.8857

Framework Versions

—Python: 3.12.2
—Sentence Transformers: 3.4.1
—Transformers: 4.50.0
—PyTorch: 2.4.1
—Accelerate: 1.4.0
—Datasets: 2.19.2
—Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

bibtex

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->