pameydorke/arcanum-quests-base-retriever-v3

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the arcanum-quests-queries-synthetic-v2 dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: BAAI/bge-m3 <!-- at revision 5617a9f61b028005a4858fdac845db406aefb181 -->
• —Maximum Sequence Length: 8192 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity
• —Supported Modality: Text
• —Training Dataset:
• —arcanum-quests-queries-synthetic-v2 <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pameydorke/arcanum-quests-base-retriever-v3")
# Run inference
sentences = [
    'need help with torian kels curse',
    "Free Torian Kel from the Curse. Northeast of Tarant there are some ancient ruins. On the west end of the dungeon you will find the Torian Kel key. Use the key to open the door on the east end of the dungeon and speak to the crumbling skeleton. It will ask you to bring it some dragon's blood. It will mark the location of where to find some on your map. Go to the Dungeon of the Dragon Pool to the northwest. Click on the pool on the 2nd level of the dungeon to receive a Vial of Dragon's Blood. Return to the Ancient Temple and use the Vial of Dragon's Blood on the crumbling skeleton to release Torian Kel from the curse. He will tell you his story and may accept an offer to join you.",
    'Clean Up the Boil. Chairman Willoughsby wishes to clean up The Boil by ridding it of the the bulk of its criminal element. The player might be able to become involved',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6918, 0.2949],
#         [0.6918, 1.0000, 0.3334],
#         [0.2949, 0.3334, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: arcanum-test
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

arcanum-quests-queries-synthetic-v2

• —Dataset: arcanum-quests-queries-synthetic-v2 at d24544a
• —Size: 912 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 912 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 5 tokens</li><li>mean: 9.62 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 180.92 tokens</li><li>max: 1044 tokens</li></ul> |
• —Samples: | anchor | positive | |:------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>kill king caladon guide</code> | <code>Kill the King of Caladon. After talking with Mr Willoughsby, a man, Heinrich Jenks appears in the room outside. He wants you to assassinate the King Of Caladon, he tells you to meet Vernon in the Kitchen for more details. When the job is done he will meet you again in Grant’s Tavern and give you 20,000 gold.You can tell Mr Willoughsby and he will send Lorham, his bodyguard to deal with him (this botches the quest).Travelling to Caladon, you need to talk with Renard first and then you will find Vernon who gives you a key.You can also get into the palace by hiding in a crate on the Caladon waterfront.Upstairs you will find a guard patrolling, he will try and kill you on sight (getting past him depends on your style prowling and backstab work well here). Make sure you loot the rooms as there is a lot of jewellery. Go through the ventilation system and explore.It is worthwhile making your way to the treasure vault. The door can be difficult to pick (or break down without damaging weapons)....</code> | | <code>gambliing quest walkthrough</code> | <code>Skill mastery quest. Gambling mastery. Gamble with Gurin Rockharrow. Women are refused entry to the Gentlemen's Club, but they still have a few avenues: Pick Pocket the key from the doorman Use Unlocking Cantrip to unbar the door Find out the name of the Club Owner from the doorman, getting his address from the Hall of Records, and then paying him a visit Once you meet with Wendell Wellington the options branch further: Pay a bribe and sleep with him for a special invitation Pick Pocket the invite Kill him and his bodyguard and loot his corpse for the invite With 8 points (2 ranks) of melee, you can threaten him for it With 4 points (1 rank) of persuasion and a Reaction Modifier of 81 you can sweet talk him for it Using this invitation will allow a lady to come and go as she pleases</code> | | <code>where is the key for big warehouse</code> | <code>Rid Mr. Plough's Warehouses of Rats. Talk to Simon Plough at the warehouse entrance and ask about his problem. Visit the smaller warehouse to the left first and kill the rats inside. You will also find a key in one of the barrels. Use the key to enter the bigger warehouse and kill everything inside. Return to Simon to finish the quest.</code> |
• —Loss: <code>GISTEmbedLoss</code> with these parameters:

  {
      "guide": "SentenceTransformer('BAAI/bge-m3')",
      "temperature": 0.01,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —warmup_steps: 0.1
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: None
• —warmup_ratio: None
• —warmup_steps: 0.1
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —enable_jit_checkpoint: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —use_cpu: False
• —seed: 42
• —data_seed: None
• —bf16: False
• —fp16: False
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: -1
• —ddp_backend: None
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —group_by_length: False
• —length_column_name: length
• —project: huggingface
• —trackio_space_id: trackio
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —auto_find_batch_size: False
• —full_determinism: False
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_num_input_tokens_seen: no
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: True
• —use_cache: False
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Training Time

• —Training: 11.6 minutes

Framework Versions

• —Python: 3.12.13
• —Sentence Transformers: 5.4.0
• —Transformers: 5.0.0
• —PyTorch: 2.10.0+cu128
• —Accelerate: 1.13.0
• —Datasets: 4.0.0
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->