danielpark/sentence-transformers-all-mini-lm-l6-v2

Sentence Transformers

We are forking sentence-transformers/all-MiniLM-L6-v2 as it is similar to the targeting dataset and use case. For more details, please check the pre-trained model weight repository.

• —https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
• —Commit Hash: 7dbbc90392e2f80f3d3c277d6e90027e55de9125

Fine-tuning

• —Fine-tune the model using a contrastive objective.
• —Compute the cosine similarity from each possible sentence pairs from the batch.
• —Then apply the cross entropy loss by comparing with true pairs.

Hyper parameters

• —Train the model during 100k steps using a batch size of 1024 (128 per TPU core).
• —Use a learning rate warm up of 500.
• —The sequence length was limited to 128 tokens.
• —Used the AdamW optimizer with a 2e-5 learning rate.
• —The full training script is accessible in this current repository: train_script.py.

Performance

Datasets