Modelpublic

matt-wisdom/KEmbed-naija-v1

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes57downloads

Model Card

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: BAAI/bge-m3
—Maximum Sequence Length: 256 tokens
—Output Dimensionality: 1024 dimensions
—Similarity Function: Cosine Similarity
—Supported Modality: Text

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What position did Bukola Saraki hold in the Nigerian Senate?',
    "Sinima to gun ni ọrọ oṣelu Naijiria, ko fẹ ẹ ni opin: Nibi ti ọrọ de e duro yii, ko si ẹni to le sọ pe ibi kan ni yoo ja si nigbẹyin. Nigba ti awuyewuye yii bẹrẹ, Adams Oshiomole lo dipo alaga ẹgbẹ APC mu. Ni ba ti se n sọrọ yi, ile ẹjọ giga ti yẹ aga nidi Oshiomole ti awọn miran si ti n ja witiwiti lati wa ni ipo alaga. Labẹ pe ibẹrẹ kọ l'onisẹ, afi ẹni ba laa ja, awọn ti o fi ẹgbẹ oselu APC silẹ fun Oshiomole bi Aarẹ ile asofin agba tẹlẹ ri Bukola Saraki ni wọn wa n gba Godwin Obaseki lalejo ninu ẹgbẹ oselu PDP bayi. Njẹ ajọsepọ awọn to kora jọ lati doju ija kọ ara wọn wọnyi, yoo jẹ ọlọjọ pipẹ tabi awọn naa yoo pada wa tutọ si ara wọn loju? Oju re e, iran ree lọrọ to wa nilẹ yi.",
    'Ọjọ Aje, Mọnde, ọsẹ yii, nijọba ipinlẹ Kwara, latọwọ Kọmisanna to n ri sọrọ ijọba ibilẹ ati lọba lọba, Aliyu Saifudeen, buwọ lu Ismail Yahaya Alebiosu gẹgẹ bii Olupo ti ilu Ajasẹ-Ipo tuntun. Kọmiṣanna sọ pe, awọn yan Ismall Yahaya Alebiosu gẹgẹ bii Olupo, nitori oun lo tọ si ipo naa. O fi kun un pe gomina fọwọ si iyansipo rẹ, ti yoo si maa ṣe atilẹyin fun un nigbakugba ti ọba alaye naa ba nilo iranlọwọ ijọba. Nigba ti Ọba Alebiosu n tẹwọ gba iwe iyansipo rẹ, o dupẹ lọwọ Gomina ipinlẹ Kwara, Abdulrahman Abdulrasaq, fun bo ṣe tẹle ohun ti awọn araalu n fẹ, to si buwọ lu iyansipo oun, o waa jẹjẹẹ pe oun ko ni i kuna lati maa ṣe ojuse oun, bẹẹ lo ṣeleri pe oun ko ni i ja awọn eeyan ilu naa kulẹ nigba kankan.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4805, 0.3301],
#         [0.4805, 1.0000, 0.5117],
#         [0.3301, 0.5117, 0.9961]], dtype=torch.bfloat16)

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Triplet

—Dataset: nigerian-val
—Evaluated with <code>TripletEvaluator</code>

Metric	Value
cosine_accuracy	0.7925

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

—Size: 4,466 training samples
—Columns: <code>anchor</code> and <code>positive</code>
—Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 6 tokens</li><li>mean: 16.51 tokens</li><li>max: 79 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 141.29 tokens</li><li>max: 256 tokens</li></ul> |
—Samples: | anchor | positive | |:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>Where is Cosmas Maduka from?</code> | <code>Maduka, an Igbo Christian from Nnewi, Anambra State was born into the family of Mr. Peter and Mrs. Rose Maduka in the City of Jos. Maduka began his entrepreneurial journey at the age of six, two years after his father died. He dropped out of primary school and started hawking Akara, a popular Nigerian food staple made from beans to support his mother.</code> | | <code>Which year was Mayotte created?</code> | <code>Mayotte broke away from the Comoros and remained with France after the Comoros declared its independence following in the 1974 referendum. Mayotte became an overseas department on 31 March 2011 and became an outermost region of the European Union on 1 January 2014, following a March 2009 referendum with an overwhelming result in favour of the department status.</code> | | <code>Ọdun wo ni Deborah Oluwaseyi Joshua gbe awo orin rẹ akọkọ jade?</code> | <code>In July 2015, she signed a record deal with Island Records. In 2015, she signed a two-year endorsement deal with Pepsi. Shay released her debut studio album Seyi or Shay in November 2015.</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

Evaluation Dataset

Unnamed Dataset

—Size: 567 evaluation samples
—Columns: <code>anchor</code> and <code>positive</code>
—Approximate statistics based on the first 567 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 6 tokens</li><li>mean: 16.5 tokens</li><li>max: 51 tokens</li></ul> | <ul><li>min: 10 tokens</li><li>mean: 138.19 tokens</li><li>max: 256 tokens</li></ul> |
—Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | <code>A cikin wace jami'a ce Manouchka Kelly Labouba ta kammala digirinta na PhD?</code> | <code>In 2005, she studied Film & Media Studies at University of California, Santa Barbara. Then, in 2007, he obtained Master de Recherche, Arts from Bordeaux Montaigne University. Later he graduated with a M.A. Critical Studies and then Ph.D. Cinema and Media Studies from University of Southern California to obtain and also graduated with a Certificate in the Business of Entertainment.</code> | | <code>Who represented Kaduna South in the 2019 Nigerian Senate?</code> | <code>The 2019 Nigerian Senate election in Kaduna State held on February 23, 2019, to elect members of the Nigerian Senate to represent Kaduna State. Kwari Suleiman Abdu representing Kaduna North, Sani Uba representing Kaduna Central both won on the platform of All Progressives Congress. while Laah Danjuma Tella representing Kaduna South returned to parliament on the platform of People's Democratic Party Results.</code> | | <code>Who is the former governor of Lagos state that also ran for Nigerian President?</code> | <code>Aare Muhammadu Buhari ti ki Asiwaju Bola Ahmed Tinubu u oriire, bee lo si ke si gbogbo awon omo Naijiria ki won fowosowopo pelu aare adiboyan naa. Buhari ni, ki awon ti ibo naa ko dun mo ninu gba ile ejo lo, ki won loo rojo nibe, nitori ohun to sele ninu ibo yii fihan pe ko si mago-mago kankan. O ni, Tinubu funra re padanu ipinle Eko, tii se ipinle re, bee loun naa padanu ipinle oun, tii se ipinle Katsina, eyi si fihan pe ko si eru kan ninu eto idibo naa, eni tawon eeyan fe lo wole ibo. O loun setan ati seto igbejoba pada fun Tinubu atawon eeyan re ko too di pe ojo igbejobo sile de ninu osu karun-un, odun yii.</code> |
—Loss: <code>MatryoshkaLoss</code> with these parameters:

json

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

Training Hyperparameters

Non-Default Hyperparameters

—eval_strategy: steps
—gradient_accumulation_steps: 16
—learning_rate: 5e-06
—num_train_epochs: 1
—lr_scheduler_type: cosine
—warmup_steps: 0.05
—bf16: True
—dataloader_num_workers: 2
—load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

—do_predict: False
—eval_strategy: steps
—prediction_loss_only: True
—per_device_train_batch_size: 8
—per_device_eval_batch_size: 8
—gradient_accumulation_steps: 16
—eval_accumulation_steps: None
—torch_empty_cache_steps: None
—learning_rate: 5e-06
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—max_grad_norm: 1.0
—num_train_epochs: 1
—max_steps: -1
—lr_scheduler_type: cosine
—lr_scheduler_kwargs: None
—warmup_ratio: None
—warmup_steps: 0.05
—log_level: passive
—log_level_replica: warning
—log_on_each_node: True
—logging_nan_inf_filter: True
—enable_jit_checkpoint: False
—save_on_each_node: False
—save_only_model: False
—restore_callback_states_from_checkpoint: False
—use_cpu: False
—seed: 42
—data_seed: None
—bf16: True
—fp16: False
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—local_rank: -1
—ddp_backend: None
—debug: []
—dataloader_drop_last: False
—dataloader_num_workers: 2
—dataloader_prefetch_factor: None
—disable_tqdm: False
—remove_unused_columns: True
—label_names: None
—load_best_model_at_end: True
—ignore_data_skip: False
—fsdp: []
—fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—deepspeed: None
—label_smoothing_factor: 0.0
—optim: adamwtorchfused
—optim_args: None
—group_by_length: False
—length_column_name: length
—project: huggingface
—trackio_space_id: trackio
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—skip_memory_metrics: True
—push_to_hub: False
—resume_from_checkpoint: None
—hub_model_id: None
—hub_strategy: every_save
—hub_private_repo: None
—hub_always_push: False
—hub_revision: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—include_for_metrics: []
—eval_do_concat_batches: True
—auto_find_batch_size: False
—full_determinism: False
—ddp_timeout: 1800
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—include_num_input_tokens_seen: no
—neftune_noise_alpha: None
—optim_target_modules: None
—batch_eval_metrics: False
—eval_on_start: False
—use_liger_kernel: False
—liger_kernel_config: None
—eval_use_gather_object: False
—average_tokens_across_devices: True
—use_cache: False
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Validation Loss	nigerian-val_cosine_accuracy
0.7033	2	8.0186	0.8019
1.3516	4	5.7152	0.7925
2.0	6	5.6023	0.7925
0.9714	17	1.8886	0.7925

—The bold row denotes the saved checkpoint.

Training Time

—Training: 16.3 minutes

Framework Versions

—Python: 3.12.13
—Sentence Transformers: 5.4.0
—Transformers: 5.0.0
—PyTorch: 2.10.0+cu128
—Accelerate: 1.13.0
—Datasets: 4.8.5
—Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

bibtex

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

bibtex

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->