novelcore/model14

1---2language:3- en4library_name: sentence-transformers5tags:6- sentence-transformers7- feature-extraction8- sentence-similarity9- transformers10pipeline_tag: sentence-similarity11---12 13# msmarco-MiniLM-L12-cos-v514This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and was designed for **semantic search**. It has been trained on 500k (query, answer) pairs from the [MS MARCO Passages dataset](https://github.com/microsoft/MSMARCO-Passage-Ranking). For an introduction to semantic search, have a look at: [SBERT.net - Semantic Search](https://www.sbert.net/examples/applications/semantic-search/README.html)15 16 17## Usage (Sentence-Transformers)18Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:19 20```21pip install -U sentence-transformers22```23 24Then you can use the model like this:25```python26from sentence_transformers import SentenceTransformer, util27 28query = "How many people live in London?"29docs = ["Around 9 Million people live in London", "London is known for its financial district"]30 31#Load the model32model = SentenceTransformer('sentence-transformers/msmarco-MiniLM-L12-cos-v5')33 34#Encode query and documents35query_emb = model.encode(query)36doc_emb = model.encode(docs)37 38#Compute dot score between query and all document embeddings39scores = util.dot_score(query_emb, doc_emb)[0].cpu().tolist()40 41#Combine docs & scores42doc_score_pairs = list(zip(docs, scores))43 44#Sort by decreasing score45doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)46 47#Output passages & scores48for doc, score in doc_score_pairs:49    print(score, doc)50```51 52 53## Usage (HuggingFace Transformers)54Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the correct pooling-operation on-top of the contextualized word embeddings.55 56```python57from transformers import AutoTokenizer, AutoModel58import torch59import torch.nn.functional as F60 61#Mean Pooling - Take average of all tokens62def mean_pooling(model_output, attention_mask):63    token_embeddings = model_output.last_hidden_state #First element of model_output contains all token embeddings64    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()65    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)66 67 68#Encode text69def encode(texts):70    # Tokenize sentences71    encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')72 73    # Compute token embeddings74    with torch.no_grad():75        model_output = model(**encoded_input, return_dict=True)76 77    # Perform pooling78    embeddings = mean_pooling(model_output, encoded_input['attention_mask'])79 80    # Normalize embeddings81    embeddings = F.normalize(embeddings, p=2, dim=1)82	83    return embeddings84 85 86# Sentences we want sentence embeddings for87query = "How many people live in London?"88docs = ["Around 9 Million people live in London", "London is known for its financial district"]89 90# Load model from HuggingFace Hub91tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/msmarco-MiniLM-L12-cos-v5")92model = AutoModel.from_pretrained("sentence-transformers/msmarco-MiniLM-L12-cos-v5")93 94#Encode query and docs95query_emb = encode(query)96doc_emb = encode(docs)97 98#Compute dot score between query and all document embeddings99scores = torch.mm(query_emb, doc_emb.transpose(0, 1))[0].cpu().tolist()100 101#Combine docs & scores102doc_score_pairs = list(zip(docs, scores))103 104#Sort by decreasing score105doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)106 107#Output passages & scores108for doc, score in doc_score_pairs:109    print(score, doc)110```111 112## Technical Details113 114In the following some technical details how this model must be used:115 116| Setting | Value |117| --- | :---: |118| Dimensions | 768 |119| Produces normalized embeddings | Yes |120| Pooling-Method | Mean pooling |121| Suitable score functions | dot-product (`util.dot_score`), cosine-similarity (`util.cos_sim`), or euclidean distance |122 123Note: When loaded with `sentence-transformers`, this model produces normalized embeddings with length 1. In that case, dot-product and cosine-similarity are equivalent. dot-product is preferred as it is faster. Euclidean distance is proportional to dot-product and can also be used.124 125## Citing & Authors126 127This model was trained by [sentence-transformers](https://www.sbert.net/). 128        129If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):130```bibtex 131@inproceedings{reimers-2019-sentence-bert,132    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",133    author = "Reimers, Nils and Gurevych, Iryna",134    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",135    month = "11",136    year = "2019",137    publisher = "Association for Computational Linguistics",138    url = "http://arxiv.org/abs/1908.10084",139}140```