datalyes/patembed-base_long_4096

1---2license: cc-by-nc-sa-4.03library_name: sentence-transformers4tags:5- sentence-transformers6- sentence-similarity7- feature-extraction8- patent9- embeddings10- mteb11language:12- en13pipeline_tag: sentence-similarity14---15 16# patembed-base_long_409617 18This is a **sentence-transformers** model trained specifically for **patent text embeddings**. It is part of the **PatenTEB** project, which provides state-of-the-art models for patent document understanding and retrieval.19 20**Note:** This model uses task-specific instruction prompts during inference for optimal performance.21 22## Model Details23 24- **Model Type**: Sentence Transformer25- **Base Architecture**: gte-modernbert-base with extended context26- **Parameters**: 149M27- **Number of Layers**: 2228- **Hidden Size**: 76829- **Embedding Dimension**: 76830- **Max Sequence Length**: 4096 tokens31- **Language**: English32- **License**: CC BY-NC-SA 4.033 34## Model Description35 36Extended context variant initialized from gte-modernbert-base with 4096-token context window.37 38This model is part of the **patembed family**, developed through multi-task learning on 13 training tasks from the PatenTEB benchmark. For detailed information about the training methodology, architecture, and comprehensive evaluation results, please refer to our paper.39 40 41 42## Usage43 44### Using Sentence Transformers45 46```python47from sentence_transformers import SentenceTransformer48 49# Load the model50model = SentenceTransformer('datalyes/patembed-base_long_4096')51 52# Encode patent texts53patent_texts = [54    "A method for manufacturing semiconductor devices...",55    "An apparatus for processing chemical compounds...",56]57embeddings = model.encode(patent_texts)58 59# Compute similarity60from sentence_transformers import util61similarity = util.cos_sim(embeddings[0], embeddings[1])62print(f"Similarity: {similarity.item():.4f}")63```64 65### Using Transformers66 67```python68from transformers import AutoTokenizer, AutoModel69import torch70import torch.nn.functional as F71 72# Load model and tokenizer73tokenizer = AutoTokenizer.from_pretrained('datalyes/patembed-base_long_4096')74model = AutoModel.from_pretrained('datalyes/patembed-base_long_4096')75 76def mean_pooling(model_output, attention_mask):77    token_embeddings = model_output[0]78    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()79    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)80 81# Tokenize and encode82texts = ["A method for manufacturing semiconductor devices..."]83encoded = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')84 85with torch.no_grad():86    model_output = model(**encoded)87    embeddings = mean_pooling(model_output, encoded['attention_mask'])88    embeddings = F.normalize(embeddings, p=2, dim=1)89```90 91### Patent Retrieval Example92 93```python94from sentence_transformers import SentenceTransformer, util95 96model = SentenceTransformer('datalyes/patembed-base_long_4096')97 98# Query patent99query = "Method for reducing power consumption in mobile devices"100 101# Candidate patents102candidates = [103    "A power management system for portable electronic devices...",104    "Chemical composition for battery manufacturing...",105    "Method for wireless data transmission in mobile networks...",106]107 108# Encode and retrieve109query_emb = model.encode(query)110candidate_embs = model.encode(candidates)111 112# Compute similarities113scores = util.cos_sim(query_emb, candidate_embs)[0]114 115# Get ranked results116results = [(candidates[i], scores[i].item()) for i in range(len(candidates))]117results.sort(key=lambda x: x[1], reverse=True)118 119for patent, score in results:120    print(f"Score: {score:.4f} - {patent[:100]}...")121```122 123## Intended Use124 125This model is designed for patent-specific tasks including:126- Patent search and retrieval127- Prior art search128- Patent classification and clustering129- Technology landscape analysis130 131For detailed training methodology, evaluation protocols, and performance analysis, please refer to our paper.132 133## Citation134 135If you use this model, please cite our paper:136 137```bibtex138@misc{ayaou2025patentebcomprehensivebenchmarkmodel,139      title={PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding}, 140      author={Iliass Ayaou and Denis Cavallucci},141      year={2025},142      eprint={2510.22264},143      archivePrefix={arXiv},144      primaryClass={cs.CL},145      url={https://arxiv.org/abs/2510.22264}146}147```148 149**Paper**: [PatenTEB on arXiv](https://arxiv.org/abs/2510.22264)150 151## License152 153This model is released under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)** license.154 155**Key Terms:**156- ✅ You can use, share, and adapt the model157- ✅ You must give appropriate credit158- ❌ You may not use the model for commercial purposes159- ⚠️ If you adapt or build upon this model, you must distribute under the same license160 161For full license details: https://creativecommons.org/licenses/by-nc-sa/4.0/162 163## Contact164 165- **Authors**: Iliass Ayaou, Denis Cavallucci166- **Institution**: ICUBE Laboratory, INSA Strasbourg167- **GitHub**: [PatentTEB/PatentTEB](https://github.com/iliass-y/patenteb)168- **HuggingFace**: [datalyes](https://huggingface.co/datalyes)169