pameydorke/arcanum-quests-base-retriever-v2

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the arcanum-quests-queries-synthetic dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: BAAI/bge-m3 <!-- at revision 5617a9f61b028005a4858fdac845db406aefb181 -->
• —Maximum Sequence Length: 8192 tokens
• —Output Dimensionality: 1024 dimensions
• —Similarity Function: Cosine Similarity
• —Supported Modality: Text
• —Training Dataset:
• —arcanum-quests-queries-synthetic <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pameydorke/arcanum-quests-base-retriever-v2")
# Run inference
sentences = [
    'test room 8 blocks orange red magenta blue light blue green lime yellow',
    'Masters of Magick. Master Of Conveyance. You can become the Master Of Conveyance by talking with Ve’tura and taking her test. Once you get into the test room, Spatial Distort yourself to the ring that is North West of you. This will take you to a room with 8 colored blocks. 4 in the middle (Green, Yellow, Red and Blue) and 4 on the sides of the room (Light Blue, Magenta, Orange and Lime). You have to send the middle blocks to the correct corner using Unseen Force. In the end the blocks should be in this order (Going Clockwise around the room) Orange, Red, Magenta, Blue, Light Blue, Green, Lime, and Yellow. Once this is done the boulder in the middle of the room vanishes. Walk into the portal. This takes you to another room. Walk into the portal in this room and you are done.',
    "Skill mastery quest. Spot trap mastery. Get staff of K’an T’au. This is a work in progressQuest takes place in The Castle S'nel N'fa. The Castle is surrounded by a moat which you must cross from your arrival point (A) to the Castle Entrance (B).Once inside you must travel from the start point (C) to the chest containing the Staff of K’an T’au at point (D). You can plow straight through with steady healing, or navigate the maze without tripping any traps. The staff is guarded by the undead titular mage, and the chest is magically held. There is also a trap door leading directly back to the surface so that you do not have to retrace the maze backwards.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6122, 0.3063],
#         [0.6122, 1.0000, 0.3606],
#         [0.3063, 0.3606, 1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Dataset: arcanum-test
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

arcanum-quests-queries-synthetic

• —Dataset: arcanum-quests-queries-synthetic at 9ef9a57
• —Size: 549 training samples
• —Columns: <code>anchor</code> and <code>positive</code>
• —Approximate statistics based on the first 549 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 5 tokens</li><li>mean: 9.99 tokens</li><li>max: 23 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 181.43 tokens</li><li>max: 1044 tokens</li></ul> |
• —Samples: | anchor | positive | |:------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>gnome and ring quest</code> | <code>The Main Quest. Finding out about the ring. So, you've got this ring you got from a gnome, and you want to find out about it. The first person to ask would be the gnome standing a bit to the northeast of the town well. He will ask you about the crash if you talk to him, and will be especially interested in the ring if you tell him about it. This little bastard is obviously not to be trusted, so don't give him the ring. He'll attack you for it later, but he's a wussy little gnome so you have nothing to fear. The next person to ask about the ring is Ristezze of Ristezze's Imported Goods, right next to the blacksmith. Ask him all about the ring. He will tell you everything, EXCEPT for where P. Schuyler & Sons, the store that originally sold it, is located. There's a couple of ways you can get this info from him: Trade him Bessie Toone’s boot (found in Bessie Toone’s mine) or the camera found on the body of Isaac Zapruder (crash site) for it. In the full game, I suggest you hang on to the ...</code> | | <code>sorcerous beast amulet</code> | <code>The Sorcerous Beast. From talking with the guard outside the isle of despair fort, you will learn that there is a sorcerous beast that has been threatening the locals. The guard will give you an amulet to dispose of it. Follow the shoreline until you find footprints and follow them back to its lair. The amulet increases you healing power with respect to poison but causes people to have a negative reaction to you.</code> | | <code>Find Liam Cameron</code> | <code>Find Liam Cameron. 1 - Starting Point2 - Liam's WorkshopThere are two things to loot inside: the chest at the foot of the bed (for a magickal trap and a scroll of disperse magick) and the dresser beside the bed (for Liam Cameron's Journal). The journal tells of a portal that has been releasing strange creatures into the area, and Liam's attempt to destroy it. Hang onto the journal so you can return it to Mrs. Cameron.3 - PathFollow the path away from Liam's Workshop. You'll have to fight some void creatures along the way, but eventually you'll stumble upon Liam's body. You'll find another magickal trap if you loot it. So obviously Liam didn't close the portal.West of Liam's body you'll find the portal. If you had an easy time with the void creatures along the path, then you might want to hang around the portal for a while killing void creatures (the portal will keep producing them).When you want to destroy the portal, there are two ways to do it. If you have a technological aptitude th...</code> |
• —Loss: <code>GISTEmbedLoss</code> with these parameters:

  {
      "guide": "SentenceTransformer('BAAI/bge-m3')",
      "temperature": 0.01,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —warmup_steps: 0.1
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 16
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: None
• —warmup_ratio: None
• —warmup_steps: 0.1
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —enable_jit_checkpoint: False
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —use_cpu: False
• —seed: 42
• —data_seed: None
• —bf16: False
• —fp16: False
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: -1
• —ddp_backend: None
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —parallelism_config: None
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —group_by_length: False
• —length_column_name: length
• —project: huggingface
• —trackio_space_id: trackio
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —auto_find_batch_size: False
• —full_determinism: False
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_num_input_tokens_seen: no
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: True
• —use_cache: False
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional
• —router_mapping: {}
• —learning_rate_mapping: {}

</details>

Training Logs

Training Time

• —Training: 27.4 minutes

Framework Versions

• —Python: 3.12.13
• —Sentence Transformers: 5.4.0
• —Transformers: 5.0.0
• —PyTorch: 2.10.0+cu128
• —Accelerate: 1.13.0
• —Datasets: 4.0.0
• —Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->