Modelpublic

Derify/ChemRanker-alpha-qed-sim

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes9downloads

Model Card

Derify/ChemRanker-alpha-qed-sim

This Cross Encoder is finetuned from Derify/ModChemBERT-IR-BASE using hard-negative triplets derived from Derify/pubchem_10m_genmol_similarity. Positive SMILES pairs are first filtered by quality and similarity constraints, then reduced to one strongest positive target per anchor molecule to create a high-signal training set for reranking. The model computes relevance scores for pairs of SMILES strings, enabling SMILES reranking and molecular semantic search.

For this variant, the positives are selected with a composite ranking criterion that combines high QED and similarity without an additional similarity-contribution cutoff. The quality stage uses strict inequality filtering (QED > 0.85, similarity > 0.5, with similarity also bounded below 1.0), and then keeps the top-scoring pair per anchor molecule.

Hard negatives are mined with Sentence Transformers using Derify/ChemMRL-beta as the teacher model and a TopK-PercPos-style margin setting based on NV-Retriever, with relative_margin=0.05 and max_negative_score_threshold = pos_score * percentage_margin. Training uses triplet-format samples with 5 mined negatives per anchor-positive pair and optimizes a multiple-negatives ranking objective, while reranking evaluation uses n-tuple samples with 30 mined negatives per query.

Model Details

Model Description

—Model Type: Cross Encoder
—Base model: Derify/ModChemBERT-IR-BASE
—Maximum Sequence Length: 512 tokens
—Number of Output Labels: 1 label
—Training Dataset:
—Derify/pubchem_10m_genmol_similarity Mined Hard Negatives
—License: apache-2.0

Model Sources

—Documentation: Sentence Transformers Documentation
—Documentation: Cross Encoder Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Transformers and Sentence Transformers libraries:

bash

pip install -U "transformers>=4.57.1,<5.0.0"
pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("Derify/ChemRanker-alpha-qed-sim")
# Get scores for pairs of texts
pairs = [
    ['c1snnc1C[NH2+]Cc1cc2c(s1)CCC2', 'c1snnc1CCC[NH2+]Cc1cc2c(s1)CCC2'],
    ['c1sc2c(c1-c1nc(C3CCOC3)no1)CCCC2', 'O=C([O-])Cc1noc(-c2csc3c2CCCC3)n1'],
    ['c1sc(C[NH2+]C2CC2)nc1C[NH+]1CCN2CCCC2C1', 'c1sc(C[NH2+]C2CC2)nc1C1CC([NH+]2CCN3CCCC3C2)C1'],
    ['c1sc(CC[NH+]2CCOCC2)nc1C[NH2+]C1CC1', 'CCc1nc(C[NH2+]C2CC2)cs1'],
    ['c1sc(CC2CCC[NH2+]2)nc1C1CCCO1', 'c1sc(CC2CCC[NH2+]2)nc1C1CCCC1'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'c1snnc1C[NH2+]Cc1cc2c(s1)CCC2',
    [
        'c1snnc1CCC[NH2+]Cc1cc2c(s1)CCC2',
        'O=C([O-])Cc1noc(-c2csc3c2CCCC3)n1',
        'c1sc(C[NH2+]C2CC2)nc1C1CC([NH+]2CCN3CCCC3C2)C1',
        'CCc1nc(C[NH2+]C2CC2)cs1',
        'c1sc(CC2CCC[NH2+]2)nc1C1CCCC1',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Cross Encoder Reranking

—Evaluated with <code>CrossEncoderRerankingEvaluator</code> with these parameters:

json

  {
      "at_k": 10
  }

Metric	Value
map	0.4266
mrr@10	0.671
ndcg@10	0.6901

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

GenMol Similarity Hard Negatives

—Dataset: GenMol Similarity Hard Negatives
—Size: 3,193,917 training samples
—Columns: <code>smilesa</code>, <code>smilesb</code>, and <code>negative</code>
—Approximate statistics based on the first 1000 samples: | | smilesa | smilesb | negative | | :------ | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | | type | string | string | string | | details | <ul><li>min: 19 characters</li><li>mean: 33.64 characters</li><li>max: 65 characters</li></ul> | <ul><li>min: 20 characters</li><li>mean: 34.24 characters</li><li>max: 48 characters</li></ul> | <ul><li>min: 19 characters</li><li>mean: 33.27 characters</li><li>max: 57 characters</li></ul> |
—Samples: | smilesa | smilesb | negative | | :---------------------------------------------- | :------------------------------------------------- | :------------------------------------------------- | | <code>c1sc2cc3c(cc2c1CC[NH2+]C1CC1)OCCO3</code> | <code>FC(F)(F)[NH2+]CCc1csc2cc3c(cc12)OCCO3</code> | <code>[NH3+]CCCc1cc2c(cc1C1CC1)OCO2</code> | | <code>c1sc2cc3c(cc2c1CC[NH2+]C1CC1)OCCO3</code> | <code>FC(F)(F)[NH2+]CCc1csc2cc3c(cc12)OCCO3</code> | <code>COc1cc2c(cc1C[NH2+]C1CCC1)OCO2</code> | | <code>c1sc2cc3c(cc2c1CC[NH2+]C1CC1)OCCO3</code> | <code>FC(F)(F)[NH2+]CCc1csc2cc3c(cc12)OCCO3</code> | <code>O=c1[nH]c2cc3c(cc2cc1CNC1CCCCC1)OCCO3</code> |
—Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

json

  {
      "scale": 10.0,
      "num_negatives": 4,
      "activation_fn": "torch.nn.modules.activation.Sigmoid"
  }

Evaluation Dataset

GenMol Similarity Hard Negatives

—Dataset: GenMol Similarity Hard Negatives
—Size: 165,968 evaluation samples
—Columns: <code>smilesa</code>, <code>smilesb</code>, <code>negative1</code>, <code>negative2</code>, <code>negative3</code>, <code>negative4</code>, <code>negative5</code>, <code>negative6</code>, <code>negative7</code>, <code>negative8</code>, <code>negative9</code>, <code>negative10</code>, <code>negative11</code>, <code>negative12</code>, <code>negative13</code>, <code>negative14</code>, <code>negative15</code>, <code>negative16</code>, <code>negative17</code>, <code>negative18</code>, <code>negative19</code>, <code>negative20</code>, <code>negative21</code>, <code>negative22</code>, <code>negative23</code>, <code>negative24</code>, <code>negative25</code>, <code>negative26</code>, <code>negative27</code>, <code>negative28</code>, <code>negative29</code>, and <code>negative30</code>
—Approximate statistics based on the first 1000 samples: | | smilesa | smilesb | negative1 | negative2 | negative3 | negative4 | negative5 | negative6 | negative7 | negative8 | negative9 | negative10 | negative11 | negative12 | negative13 | negative14 | negative15 | negative16 | negative17 | negative18 | negative19 | negative20 | negative21 | negative22 | negative23 | negative24 | negative25 | negative26 | negative27 | negative28 | negative29 | negative30 | | :------ | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | | type | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | | details | <ul><li>min: 17 characters</li><li>mean: 37.57 characters</li><li>max: 96 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 34.45 characters</li><li>max: 70 characters</li></ul> | <ul><li>min: 18 characters</li><li>mean: 35.67 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 12 characters</li><li>mean: 35.13 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.28 characters</li><li>max: 81 characters</li></ul> | <ul><li>min: 17 characters</li><li>mean: 35.36 characters</li><li>max: 73 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.12 characters</li><li>max: 70 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.09 characters</li><li>max: 84 characters</li></ul> | <ul><li>min: 16 characters</li><li>mean: 35.16 characters</li><li>max: 64 characters</li></ul> | <ul><li>min: 13 characters</li><li>mean: 35.26 characters</li><li>max: 90 characters</li></ul> | <ul><li>min: 11 characters</li><li>mean: 35.16 characters</li><li>max: 90 characters</li></ul> | <ul><li>min: 15 characters</li><li>mean: 35.36 characters</li><li>max: 74 characters</li></ul> | <ul><li>min: 12 characters</li><li>mean: 35.16 characters</li><li>max: 63 characters</li></ul> | <ul><li>min: 15 characters</li><li>mean: 35.51 characters</li><li>max: 73 characters</li></ul> | <ul><li>min: 13 characters</li><li>mean: 35.21 characters</li><li>max: 69 characters</li></ul> | <ul><li>min: 10 characters</li><li>mean: 34.93 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 17 characters</li><li>mean: 35.41 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 18 characters</li><li>mean: 35.1 characters</li><li>max: 72 characters</li></ul> | <ul><li>min: 15 characters</li><li>mean: 35.43 characters</li><li>max: 62 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.36 characters</li><li>max: 65 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.48 characters</li><li>max: 65 characters</li></ul> | <ul><li>min: 18 characters</li><li>mean: 35.25 characters</li><li>max: 65 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.48 characters</li><li>max: 81 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.38 characters</li><li>max: 68 characters</li></ul> | <ul><li>min: 18 characters</li><li>mean: 35.67 characters</li><li>max: 68 characters</li></ul> | <ul><li>min: 16 characters</li><li>mean: 35.53 characters</li><li>max: 67 characters</li></ul> | <ul><li>min: 14 characters</li><li>mean: 35.39 characters</li><li>max: 83 characters</li></ul> | <ul><li>min: 18 characters</li><li>mean: 35.74 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 17 characters</li><li>mean: 35.56 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 11 characters</li><li>mean: 35.37 characters</li><li>max: 64 characters</li></ul> | <ul><li>min: 16 characters</li><li>mean: 35.51 characters</li><li>max: 77 characters</li></ul> | <ul><li>min: 16 characters</li><li>mean: 35.35 characters</li><li>max: 72 characters</li></ul> |
—Samples: | smilesa | smilesb | negative1 | negative2 | negative3 | negative4 | negative5 | negative6 | negative7 | negative8 | negative9 | negative10 | negative11 | negative12 | negative13 | negative14 | negative15 | negative16 | negative17 | negative18 | negative19 | negative20 | negative21 | negative22 | negative23 | negative24 | negative25 | negative26 | negative27 | negative28 | negative29 | negative30 | | :--------------------------------------------------- | :---------------------------------------------------------- | :---------------------------------------------------------- | :---------------------------------------------------- | :------------------------------------------------- | :------------------------------------------ | :-------------------------------------------------- | :----------------------------------------- | :------------------------------------------------- | :--------------------------------------------------- | :---------------------------------------------------- | :------------------------------------------------- | :------------------------------------------------------- | :----------------------------------------------- | :------------------------------------------------------ | :------------------------------------------------ | :------------------------------------------------------ | :--------------------------------------------------- | :------------------------------------------------ | :----------------------------------------------- | :---------------------------------------------------- | :----------------------------------------------- | :----------------------------------------------------- | :-------------------------------------------------- | :----------------------------------------------------- | :------------------------------------------------- | :--------------------------------------------- | :--------------------------------------------------------- | :------------------------------------------------ | :--------------------------------------------------- | :------------------------------------------------------- | :----------------------------------------------- | | <code>c1snnc1C[NH2+]Cc1cc2c(s1)CCC2</code> | <code>c1snnc1CCC[NH2+]Cc1cc2c(s1)CCC2</code> | <code>c1snnc1CCC[NH2+]Cc1cc2c(s1)CCC2</code> | <code>Cn1cc(C[NH2+]Cc2cc3c(s2)CCC3)nn1</code> | <code>Cn1cc(CC[NH2+]Cc2cc3c(s2)CCC3)nn1</code> | <code>Cc1cc(C[NH2+]Cc2csnn2)sc1C</code> | <code>NC(=O)c1csc(C[NH2+]Cc2cc3c(s2)CCC3)c1</code> | <code>Cc1cc(CC[NH2+]Cc2csnn2)sc1C</code> | <code>N#CCc1csc(C[NH2+]Cc2cc3c(s2)CCC3)c1</code> | <code>Ic1ccc(C[NH2+]Cc2cc3c(s2)CCC3)o1</code> | <code>c1ncc(C[NH2+]Cc2csnn2)s1</code> | <code>c1c(C[NH2+]CC2CC2)sc2c1CSCC2</code> | <code>N#Cc1cc(F)cc(C[NH2+]Cc2cc3c(s2)CCC3)c1</code> | <code>c1cc(C[NH2+]Cc2nc3c(s2)CCC3)no1</code> | <code>CCc1ccc(C[NH2+]Cc2csnn2)s1</code> | <code>NCc1csc(NCc2cc3c(s2)CCC3)n1</code> | <code>CNH+Cc1nnc(-c2cc3c(s2)CCCC3)o1</code> | <code>Fc1cc(C[NH2+]Cc2cc3c(s2)CCC3)ccc1Br</code> | <code>FC(F)(F)C[NH2+]Cc1cc2c(s1)CCSC2</code> | <code>c1cc(C[NH2+]Cc2cc3c(s2)CCC3)c[nH]1</code> | <code>Cc1cc(C)c(CC[NH2+]Cc2cc3c(s2)CCC3)c(C)c1</code> | <code>Oc1ccc(C[NH2+]Cc2cc3c(s2)CCC3)cc1Br</code> | <code>O=C([O-])c1ccc(CC[NH2+]Cc2cc3c(s2)CCC3)s1</code> | <code>c1c(C[NH2+]CC2CCCC2)sc2c1CCC2</code> | <code>O=C([O-])c1ccc(C[NH2+]Cc2cc3c(s2)CCC3)s1</code> | <code>COc1cc(C)cc(C[NH2+]Cc2cc3c(s2)CCC3)c1</code> | <code>CCc1cnc(C[NH2+]Cc2csnn2)s1</code> | <code>Clc1cc(C[NH2+]Cc2cc3c(s2)CCC3)ccc1Br</code> | <code>c1c(C[NH2+]CC2CC2)sc2c1CCCCC2</code> | <code>Cc1ccccc1C[NH2+]Cc1cc2c(s1)CCC2</code> | <code>c1cc(C[NH+]2CCCC2)sc1C[NH2+]Cc1cc2c(s1)CCC2</code> | <code>Cc1cc(C[NH2+]Cc2cc3c(s2)CCC3)ccc1F</code> | | <code>c1sc2c(c1-c1nc(C3CCOC3)no1)CCCC2</code> | <code>O=C([O-])Cc1noc(-c2csc3c2CCCC3)n1</code> | <code>Nc1sc2c(c1-c1nc(C3CCOC3)no1)CCCC2</code> | <code>Nc1sc2c(c1-c1nc(C3CCC3)no1)CCCC2</code> | <code>c1c(-c2nc(C3CCCNC3)no2)sc2c1CCCCCC2</code> | <code>Nc1sccc1-c1nc(C2CCCOC2)no1</code> | <code>Nc1sc2c(c1-c1nc(C3CCCO3)no1)CCCC2</code> | <code>Cc1csc(-c2nc(C3CCOCC3)no2)c1N</code> | <code>Cc1oc2c(c1-c1nc(C3CCOC3)no1)C(=O)CCC2</code> | <code>c1c(-c2nc(C3C[NH2+]CCO3)no2)sc2c1CCCCC2</code> | <code>O=C([O-])Nc1sc2c(c1-c1nc(C3CC3)no1)CCCC2</code> | <code>c1cc2c(s1)CCCC2c1nc(C2CC2)no1</code> | <code>CC(=O)N1CCCC(c2noc(-c3cc4c(s3)CCCCCC4)n2)C1</code> | <code>Cc1cc(-c2nc([C@@H]3CCOC3)no2)c(N)s1</code> | <code>c1cc2c(nc1-c1noc(C3CCCOC3)n1)CCCC2</code> | <code>Nc1sccc1-c1nc(C2CCCC2)no1</code> | <code>c1cc2c(nc1-c1noc(C3CCOCC3)n1)CCCC2</code> | <code>[NH3+]C(c1noc(-c2cc3c(s2)CCCC3)n1)C1CC1</code> | <code>c1cc2c(c(-c3nc(C4CCOCC4)no3)c1)CCCN2</code> | <code>c1c(-c2nc(C3CC3)no2)nn2c1CCCC2</code> | <code>CN1CC(c2noc(-c3cc4c(s3)CCCC4)n2)CC1=O</code> | <code>O=C([O-])Cc1noc(-c2csc3c2CCCC3)n1</code> | <code>Oc1c(-c2nc(C3CCC(F)(F)C3)no2)ccc2c1CCCC2</code> | <code>Cc1cc(=O)c(-c2noc(C3CCCOC3)n2)c2n1CCC2</code> | <code>O=C([O-])CNc1sc2c(c1-c1nc(C3CC3)no1)CCCC2</code> | <code>CC1CCc2c(sc(N)c2-c2nc(C3CC3)no2)C1</code> | <code>Cn1nc(-c2nc(C3CCCO3)no2)c2c1CCCC2</code> | <code>O=C(Nc1sc2c(c1-c1nc(C3CC3)no1)COCC2)C1=CCCCC1</code> | <code>Cc1cscc1-c1noc(C2CCOCC2)n1</code> | <code>CC1(C)CCCc2sc(N)c(-c3nc(C4CC4)no3)c21</code> | <code>Clc1cc2c(c(-c3nc(C4CCOC4)no3)c1)OCC2</code> | <code>Nc1sc2c(c1-c1nnc(C3CC3)o1)CCCC2</code> | | <code>c1sc(C[NH2+]C2CC2)nc1C[NH+]1CCN2CCCC2C1</code> | <code>c1sc(C[NH2+]C2CC2)nc1C1CC([NH+]2CCN3CCCC3C2)C1</code> | <code>c1sc(C[NH2+]C2CC2)nc1C1CC([NH+]2CCN3CCCC3C2)C1</code> | <code>CC(C)[NH2+]Cc1nc(C[NH+]2CCC3CCCCC3C2)cs1</code> | <code>CN1C2CCC1CNH+n1)CC2</code> | <code>Nc1nc(CC[NH+]2CCCN3CCCC3C2)cs1</code> | <code>CC1CNH+n2)CCN1C</code> | <code>Oc1csc(CN2CCCC3C[NH2+]CC32)n1</code> | <code>CCc1nc(C[NH+]2CCCC3CCCCC32)cs1</code> | <code>C[NH2+]Cc1csc(N2CC[NH+]3CCCC3C2)n1</code> | <code>[NH3+]Cc1nc(C[NH+]2CCC3CCCCC32)cs1</code> | <code>CC1CN2CCCCC2C[NH+]1Cc1csc(CC[NH3+])n1</code> | <code>CCCc1nc(CN2CCCC2C2CCC[NH2+]2)cs1</code> | <code>ClCCc1nc(CN2CCCC2C2CCC[NH2+]2)cs1</code> | <code>c1cc(C[NH2+]C2CC2)c(C[NH+]2CCN3CCCCC3C2)o1</code> | <code>O=C(Cc1nc(CCl)cs1)N1CCC[NH+]2CCCC2C1</code> | <code>N#CCc1nc(C[NH+]2CCCC3CCCCC32)cs1</code> | <code>CC[NH2+]Cc1csc(N2CCC3C(CCC[NH+]3C)C2)n1</code> | <code>c1sc(C[NH2+]C2CC2)nc1C[NH+]1CCCCC1</code> | <code>[NH3+]Cc1nc(C[NH+]2CCCC2C2CCCC2)cs1</code> | <code>Cc1csc(C[NH+]2CCC3C[NH2+]CC3C2)n1</code> | <code>ClOCc1csc(C[NH+]2CC3C[NH2+]CC3C2)n1</code> | <code>c1cc(C[NH+]2CCCN3CCCC3C2)nc(C2CC2)n1</code> | <code>Cc1ccsc1C[NH2+]CCN1CCN2CCCC2C1</code> | <code>c1sc(C[NH2+]C2CCCC2)nc1C[NH+]1CCCCC1</code> | <code>Brc1csc(C[NH2+]CCN2CCN3CCCCC3C2)c1</code> | <code>Cc1nc(CCC[NH2+]C2CCN3CCCCC23)cs1</code> | <code>CCOC(=O)c1nc(CN2CC3CCC[NH2+]C3C2)cs1</code> | <code>CCCC(=O)c1nc(CN2CC3CCC[NH2+]C3C2)cs1</code> | <code>CC(C)(C)c1csc(CN2CCC[NH2+]C(C3CC3)C2)n1</code> | <code>COCc1nc(CN2CCC([NH3+])C2)cs1</code> | <code>CCC[NH2+]Cc1nc(C[NH+]2CC3CCC2C3)cs1</code> |
—Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

json

  {
      "scale": 10.0,
      "num_negatives": 4,
      "activation_fn": "torch.nn.modules.activation.Sigmoid"
  }

Training Hyperparameters

Non-Default Hyperparameters

—eval_strategy: epoch
—per_device_train_batch_size: 256
—per_device_eval_batch_size: 256
—torch_empty_cache_steps: 1000
—learning_rate: 3e-05
—weight_decay: 1e-05
—max_grad_norm: None
—lr_scheduler_type: warmupstabledecay
—lr_scheduler_kwargs: {'numdecaysteps': 6238, 'warmuptype': 'linear', 'decaytype': '1-sqrt'}
—warmup_steps: 6238
—seed: 12
—data_seed: 24681357
—bf16: True
—bf16_full_eval: True
—tf32: True
—dataloader_num_workers: 8
—dataloader_prefetch_factor: 2
—load_best_model_at_end: True
—optim: stable_adamw
—optim_args: decouplelr=True,maxlr=3e-05
—dataloader_persistent_workers: True
—resume_from_checkpoint: False
—gradient_checkpointing: True
—torch_compile: True
—torch_compile_backend: inductor
—torch_compile_mode: max-autotune
—eval_on_start: True
—batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

—overwrite_output_dir: False
—do_predict: False
—eval_strategy: epoch
—prediction_loss_only: True
—per_device_train_batch_size: 256
—per_device_eval_batch_size: 256
—per_gpu_train_batch_size: None
—per_gpu_eval_batch_size: None
—gradient_accumulation_steps: 1
—eval_accumulation_steps: None
—torch_empty_cache_steps: 1000
—learning_rate: 3e-05
—weight_decay: 1e-05
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—max_grad_norm: None
—num_train_epochs: 3
—max_steps: -1
—lr_scheduler_type: warmupstabledecay
—lr_scheduler_kwargs: {'numdecaysteps': 6238, 'warmuptype': 'linear', 'decaytype': '1-sqrt'}
—warmup_ratio: 0.0
—warmup_steps: 6238
—log_level: passive
—log_level_replica: warning
—log_on_each_node: True
—logging_nan_inf_filter: True
—save_safetensors: True
—save_on_each_node: False
—save_only_model: False
—restore_callback_states_from_checkpoint: False
—no_cuda: False
—use_cpu: False
—use_mps_device: False
—seed: 12
—data_seed: 24681357
—jit_mode_eval: False
—bf16: True
—fp16: False
—fp16_opt_level: O1
—half_precision_backend: auto
—bf16_full_eval: True
—fp16_full_eval: False
—tf32: True
—local_rank: 0
—ddp_backend: None
—tpu_num_cores: None
—tpu_metrics_debug: False
—debug: []
—dataloader_drop_last: True
—dataloader_num_workers: 8
—dataloader_prefetch_factor: 2
—past_index: -1
—disable_tqdm: False
—remove_unused_columns: True
—label_names: None
—load_best_model_at_end: True
—ignore_data_skip: False
—fsdp: []
—fsdp_min_num_params: 0
—fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
—fsdp_transformer_layer_cls_to_wrap: None
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—deepspeed: None
—label_smoothing_factor: 0.0
—optim: stable_adamw
—optim_args: decouplelr=True,maxlr=3e-05
—adafactor: False
—group_by_length: False
—length_column_name: length
—project: huggingface
—trackio_space_id: trackio
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—dataloader_pin_memory: True
—dataloader_persistent_workers: True
—skip_memory_metrics: True
—use_legacy_prediction_loop: False
—push_to_hub: False
—resume_from_checkpoint: False
—hub_model_id: None
—hub_strategy: every_save
—hub_private_repo: None
—hub_always_push: False
—hub_revision: None
—gradient_checkpointing: True
—gradient_checkpointing_kwargs: None
—include_inputs_for_metrics: False
—include_for_metrics: []
—eval_do_concat_batches: True
—fp16_backend: auto
—push_to_hub_model_id: None
—push_to_hub_organization: None
—mp_parameters:
—auto_find_batch_size: False
—full_determinism: False
—torchdynamo: None
—ray_scope: last
—ddp_timeout: 1800
—torch_compile: True
—torch_compile_backend: inductor
—torch_compile_mode: max-autotune
—include_tokens_per_second: False
—include_num_input_tokens_seen: no
—neftune_noise_alpha: None
—optim_target_modules: None
—batch_eval_metrics: False
—eval_on_start: True
—use_liger_kernel: False
—liger_kernel_config: None
—eval_use_gather_object: False
—average_tokens_across_devices: True
—prompts: None
—batch_sampler: no_duplicates
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss	Validation Loss	ndcg@10
0.0002	1	1.2724	-	-
0.1603	1000	0.1583	-	-
0.3206	2000	0.0196	-	-
0.4809	3000	0.0112	-	-
0.6412	4000	0.0079	-	-
0.8015	5000	0.0063	-	-
0.9618	6000	0.0053	-	-
1.0	6238	-	1.6835	0.6811
1.1222	7000	0.0045	-	-
1.2825	8000	0.0041	-	-
1.4428	9000	0.0037	-	-
1.6031	10000	0.0034	-	-
1.7634	11000	0.0032	-	-
1.9237	12000	0.003	-	-
2.0	12476	-	1.6853	0.6891
2.0840	13000	0.0028	-	-
2.2443	14000	0.0026	-	-
2.4046	15000	0.0026	-	-
2.5649	16000	0.0025	-	-
2.7252	17000	0.0024	-	-
2.8855	18000	0.0023	-	-
3.0	18714	-	1.6982	0.6901

—The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

—Energy Consumed: 17.863 kWh
—Carbon Emitted: 3.667 kg of CO2
—Hours Used: 29.477 hours

Training Hardware

—On Cloud: No
—GPU Model: 2 x NVIDIA GeForce RTX 3090
—CPU Model: AMD Ryzen 7 3700X 8-Core Processor
—RAM Size: 62.70 GB

Framework Versions

—Python: 3.13.7
—Sentence Transformers: 5.1.2
—Transformers: 4.57.1
—PyTorch: 2.9.0+cu128
—Accelerate: 1.11.0
—Datasets: 4.4.1
—Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

NV-Retriever

bibtex

@misc{moreira2025nvretrieverimprovingtextembedding,
      title={NV-Retriever: Improving text embedding models with effective hard-negative mining}, 
      author={Gabriel de Souza P. Moreira and Radek Osmulski and Mengyao Xu and Ronay Ak and Benedikt Schifferer and Even Oldridge},
      year={2025},
      eprint={2407.15831},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2407.15831}, 
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->