Lihuchen/pearl_small
🦪⚪ PEARL-small
Learning High-Quality and General-Purpose Phrase Representations. <br> Lihu Chen, Gaël Varoquaux, Fabian M. Suchanek. Accepted by EACL Findings 2024 <br>
PEARL-small is a lightweight string embedding model. It is the tool of choice for semantic similarity computation for strings, creating excellent embeddings for string matching, entity retrieval, entity clustering, fuzzy join... <br> It differs from typical sentence embedders because it incorporates phrase type information and morphological features, allowing it to better capture variations in strings. The model is a variant of E5-small finetuned on our constructed context-free dataset to yield better representations for phrases and strings. <br>
🤗 PEARL-small 🤗 PEARL-base 📐 PEARL Benchmark 🏆 PEARL Leaderboard <br>
Cost comparison of FastText and PEARL. The estimated memory is calculated by the number of parameters (float16). The unit of inference speed is *ms/512 samples. The FastText model here is crawl-300d-2M-subword.bin. | Model |Avg Score| Estimated Memory |Speed GPU | Speed CPU | |-|-|-|-|-| |FastText|40.3|1200MB|-|57ms| |PEARL-small|62.5|68MB|42ms|446ms| |PEARL-base|64.8|220MB|89ms|1394ms|
Usage
Sentence Transformers
PEARL is integrated with the Sentence Transformers library (Thanks for Tom Aarsen's contribution), and can be used like so:
from sentence_transformers import SentenceTransformer, util
query_texts = ["The New York Times"]
doc_texts = [ "NYTimes", "New York Post", "New York"]
input_texts = query_texts + doc_texts
model = SentenceTransformer("Lihuchen/pearl_small")
embeddings = model.encode(input_texts)
scores = util.cos_sim(embeddings[0], embeddings[1:]) * 100
print(scores.tolist())
# [[90.56318664550781, 79.65763854980469, 75.52056121826172]]Transformers
You can also use transformers to use PEARL. Below is an example of entity retrieval, and we reuse the code from E5.
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
def encode_text(model, input_texts):
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
return embeddings
query_texts = ["The New York Times"]
doc_texts = [ "NYTimes", "New York Post", "New York"]
input_texts = query_texts + doc_texts
tokenizer = AutoTokenizer.from_pretrained('Lihuchen/pearl_small')
model = AutoModel.from_pretrained('Lihuchen/pearl_small')
# encode
embeddings = encode_text(model, input_texts)
# calculate similarity
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())
# expected outputs
# [[90.56318664550781, 79.65763854980469, 75.52054595947266]]Training and Evaluation
Have a look at our code on Github
Citation
If you find our work useful, please give us a citation:
@inproceedings{chen2024learning,
title={Learning High-Quality and General-Purpose Phrase Representations},
author={Chen, Lihu and Varoquaux, Gael and Suchanek, Fabian},
booktitle={Findings of the Association for Computational Linguistics: EACL 2024},
pages={983--994},
year={2024}
}