BM-K/KoSimCSE-roberta-multitask

1---2language: ko3tags:4  - korean5---6 7https://github.com/BM-K/Sentence-Embedding-is-all-you-need8 9# Korean-Sentence-Embedding10🍭 Korean sentence embedding repository. You can download the pre-trained models and inference right away, also it provides environments where individuals can train models.11 12## Quick tour13```python14import torch15from transformers import AutoModel, AutoTokenizer16 17def cal_score(a, b):18    if len(a.shape) == 1: a = a.unsqueeze(0)19    if len(b.shape) == 1: b = b.unsqueeze(0)20 21    a_norm = a / a.norm(dim=1)[:, None]22    b_norm = b / b.norm(dim=1)[:, None]23    return torch.mm(a_norm, b_norm.transpose(0, 1)) * 10024 25model = AutoModel.from_pretrained('BM-K/KoSimCSE-roberta-multitask') 26AutoTokenizer.from_pretrained('BM-K/KoSimCSE-roberta-multitask')27 28sentences = ['치타가 들판을 가로 질러 먹이를 쫓는다.',29             '치타 한 마리가 먹이 뒤에서 달리고 있다.',30             '원숭이 한 마리가 드럼을 연주한다.']31 32inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")33embeddings, _ = model(**inputs, return_dict=False)34 35score01 = cal_score(embeddings[0][0], embeddings[1][0])36score02 = cal_score(embeddings[0][0], embeddings[2][0])37```38 39## Performance40- Semantic Textual Similarity test set results <br>41 42| Model                  | AVG | Cosine Pearson | Cosine Spearman | Euclidean Pearson | Euclidean Spearman | Manhattan Pearson | Manhattan Spearman | Dot Pearson | Dot Spearman |43|------------------------|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|44| KoSBERT<sup>†</sup><sub>SKT</sub>    | 77.40 | 78.81 | 78.47 | 77.68 | 77.78 | 77.71 | 77.83 | 75.75 | 75.22 |45| KoSBERT              | 80.39 | 82.13 | 82.25 | 80.67 | 80.75 | 80.69 | 80.78 | 77.96 | 77.90 |46| KoSRoBERTa           | 81.64 | 81.20 | 82.20 | 81.79 | 82.34 | 81.59 | 82.20 | 80.62 | 81.25 |47| | | | | | | | | |48| KoSentenceBART         | 77.14 | 79.71 | 78.74 | 78.42 | 78.02 | 78.40 | 78.00 | 74.24 | 72.15 |49| KoSentenceT5          | 77.83 | 80.87 | 79.74 | 80.24 | 79.36 | 80.19 | 79.27 | 72.81 | 70.17 |50| | | | | | | | | |51| KoSimCSE-BERT<sup>†</sup><sub>SKT</sub>   | 81.32 | 82.12 | 82.56 | 81.84 | 81.63 | 81.99 | 81.74 | 79.55 | 79.19 |52| KoSimCSE-BERT              | 83.37 | 83.22 | 83.58 | 83.24 | 83.60 | 83.15 | 83.54 | 83.13 | 83.49 |53| KoSimCSE-RoBERTa          | 83.65 | 83.60 | 83.77 | 83.54 | 83.76 | 83.55 | 83.77 | 83.55 | 83.64 |54| | | | | | | | | | |55| KoSimCSE-BERT-multitask              | 85.71 | 85.29 | 86.02 | 85.63 | 86.01 | 85.57 | 85.97 | 85.26 | 85.93 |56| KoSimCSE-RoBERTa-multitask          | 85.77 | 85.08 | 86.12 | 85.84 | 86.12 | 85.83 | 86.12 | 85.03 | 85.99 |