pameydorke/arcanum-quests-base-retriever

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the arcanum-quests-queries-synthetic dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: BAAI/bge-m3 <!-- at revision 5617a9f61b028005a4858fdac845db406aefb181 -->
• —Maximum Sequence Length: 8192 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity
• —Supported Modality: Text
• —Training Dataset:
• —arcanum-quests-queries-synthetic <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pameydorke/arcanum-quests-base-retriever")
# Run inference
sentences = [
    'whiskey pickup tarant',
    "The Boil When chatting with Caleb Malloy in his pub, he will explain the geography of The Boil and how the gangs (or Clans) operate. If the player is unwilling to commit to joining one of the gangs, then Caleb can offer them some neutral work This is to retrieve a scheduled whiskey delivery that has been getting waylaid of late by one of the gangs Caleb's delivery boy is stalled on Garillon Bridge, so the party needs to go there to make the pickup Garillon Bridge The player can find Caleb's boy (actually Dwarf) on the bridge between The Boil and Tarant The party will get jumped by a rival gang as they approach. This may happen before or after actually getting the whiskey. Regardless of order, dispatch the gang and take possession of the whiskey The Boil Return to the pub and give Caleb the Whiskey He will give you his thanks, and a reward of 500 gp and 3100 exp",
    "Gladys in Dernholm will tell you that she wants a ring that used to belong to her son. Speak with Archibald by the dock and ask him about the ring. He is unwilling to hand it over to you. Talk to his son Bernard (just outside the house-on the docks), he will tell you its some sort of game between the two. Glady sends someone looking for the ring every so often. The ring belongs to Gladys son, who gave it to Bernard sister. Hints to a possible romantic connection between the two. Option 1: Pickpocket the ring from Archibald. Tell Glady he gave it to you. Then you'll receive 800 experience and some alignment. Option 2: Kill Archibald and Bernard for the ring, Gladys will be very upset with you. No reward. Option 3: Use Persuasion (at least 1+ point in skill) and talk to Bernard and Archibald to hand over the ring. Go back to Archibald and tell him you think Gladys is attracted to him, and he'll give you the ring. Then when you return it to Gladys you'll receive 1600 experience and a boost to your alignment.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5063, 0.2837],
#         [0.5063, 1.0000, 0.4145],
#         [0.2837, 0.4145, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: arcanum-test
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

arcanum-quests-queries-synthetic

• —Dataset: arcanum-quests-queries-synthetic at 69805bb
• —Size: 549 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 549 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 5 tokens</li><li>mean: 9.99 tokens</li><li>max: 23 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 171.09 tokens</li><li>max: 1037 tokens</li></ul> |
• —Samples: | anchor | positive | |:------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>gnome and ring quest</code> | <code>So, you've got this ring you got from a gnome, and you want to find out about it. The first person to ask would be the gnome standing a bit to the northeast of the town well. He will ask you about the crash if you talk to him, and will be especially interested in the ring if you tell him about it. This little bastard is obviously not to be trusted, so don't give him the ring. He'll attack you for it later, but he's a wussy little gnome so you have nothing to fear. The next person to ask about the ring is Ristezze of Ristezze's Imported Goods, right next to the blacksmith. Ask him all about the ring. He will tell you everything, EXCEPT for where P. Schuyler & Sons, the store that originally sold it, is located. There's a couple of ways you can get this info from him: Trade him Bessie Toone’s boot (found in Bessie Toone’s mine) or the camera found on the body of Isaac Zapruder (crash site) for it. In the full game, I suggest you hang on to the camera, as it might help later on. A charism...</code> | | <code>sorcerous beast amulet</code> | <code>From talking with the guard outside the isle of despair fort, you will learn that there is a sorcerous beast that has been threatening the locals. The guard will give you an amulet to dispose of it. Follow the shoreline until you find footprints and follow them back to its lair. The amulet increases you healing power with respect to poison but causes people to have a negative reaction to you.</code> | | <code>Find Liam Cameron</code> | <code>1 - Starting Point2 - Liam's WorkshopThere are two things to loot inside: the chest at the foot of the bed (for a magickal trap and a scroll of disperse magick) and the dresser beside the bed (for Liam Cameron's Journal). The journal tells of a portal that has been releasing strange creatures into the area, and Liam's attempt to destroy it. Hang onto the journal so you can return it to Mrs. Cameron.3 - PathFollow the path away from Liam's Workshop. You'll have to fight some void creatures along the way, but eventually you'll stumble upon Liam's body. You'll find another magickal trap if you loot it. So obviously Liam didn't close the portal.West of Liam's body you'll find the portal. If you had an easy time with the void creatures along the path, then you might want to hang around the portal for a while killing void creatures (the portal will keep producing them).When you want to destroy the portal, there are two ways to do it. If you have a technological aptitude then you should use t...</code> |
• —Loss: <code>GISTEmbedLoss</code> with these parameters:

  {
      "guide": "SentenceTransformer('BAAI/bge-m3')",
      "temperature": 0.01,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 4
• —per_device_eval_batch_size: 4
• —gradient_accumulation_steps: 4
• —warmup_steps: 0.1
• —fp16: True
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 4
• —per_device_eval_batch_size: 4
• —gradient_accumulation_steps: 4
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: None
• —warmup_ratio: None
• —warmup_steps: 0.1
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —enable_jit_checkpoint: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —use_cpu: False
• —seed: 42
• —data_seed: None
• —bf16: False
• —fp16: True
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: -1
• —ddp_backend: None
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —group_by_length: False
• —length_column_name: length
• —project: huggingface
• —trackio_space_id: trackio
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —auto_find_batch_size: False
• —full_determinism: False
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_num_input_tokens_seen: no
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: True
• —use_cache: False
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Training Time

• —Training: 11.4 minutes

Framework Versions

• —Python: 3.12.13
• —Sentence Transformers: 5.4.0
• —Transformers: 5.0.0
• —PyTorch: 2.10.0+cu128
• —Accelerate: 1.13.0
• —Datasets: 4.0.0
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->