AnnaWegmann/Style-Embedding

1---2pipeline_tag: sentence-similarity3tags:4- sentence-transformers5- feature-extraction6- sentence-similarity7- transformers8datasets:9- AnnaWegmann/StyleEmbeddingData10base_model:11- FacebookAI/roberta-base12---13 14# Style Embedding15 16This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.17 18for more info see [Style-Embeddings](https://github.com/nlpsoc/Style-Embeddings)19 20see published paper at [https://aclanthology.org/2022.repl4nlp-1.26/](https://aclanthology.org/2022.repl4nlp-1.26/) and arxiv paper at [https://arxiv.org/abs/2204.04907](https://arxiv.org/abs/2204.04907).21 22## Usage (Sentence-Transformers)23 24Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:25 26```27pip install -U sentence-transformers28```29 30Then you can use the model like this:31 32```python33from sentence_transformers import SentenceTransformer34sentences = ["This is an example sentence", "Each sentence is converted"]35 36model = SentenceTransformer('{MODEL_NAME}')37embeddings = model.encode(sentences)38print(embeddings)39```40 41 42 43## Usage (HuggingFace Transformers)44Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.45 46```python47from transformers import AutoTokenizer, AutoModel48import torch49 50 51#Mean Pooling - Take attention mask into account for correct averaging52def mean_pooling(model_output, attention_mask):53    token_embeddings = model_output[0] #First element of model_output contains all token embeddings54    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()55    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)56 57 58# Sentences we want sentence embeddings for59sentences = ['This is an example sentence', 'Each sentence is converted']60 61# Load model from HuggingFace Hub62tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')63model = AutoModel.from_pretrained('{MODEL_NAME}')64 65# Tokenize sentences66encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')67 68# Compute token embeddings69with torch.no_grad():70    model_output = model(**encoded_input)71 72# Perform pooling. In this case, mean pooling.73sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])74 75print("Sentence embeddings:")76print(sentence_embeddings)77```78 79 80 81## Evaluation Results82 83<!--- Describe how your model was evaluated -->84 85For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})86 87 88## Training89The model was trained with the parameters:90 91**DataLoader**:92 93`torch.utils.data.dataloader.DataLoader` of length 26250 with parameters:94```95{'batch_size': 8, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}96```97 98**Loss**:99 100`sentence_transformers.losses.TripletLoss.TripletLoss` with parameters:101  ```102  {'distance_metric': 'TripletDistanceMetric.COSINE', 'triplet_margin': 0.5}103  ```104 105Parameters of the fit()-Method:106```107{108    "epochs": 4,109    "evaluation_steps": 0,110    "evaluator": "sentence_transformers.evaluation.TripletEvaluator.TripletEvaluator",111    "max_grad_norm": 1,112    "optimizer_class": "<class 'transformers.optimization.AdamW'>",113    "optimizer_params": {114        "correct_bias": true,115        "eps": 1e-08,116        "lr": 2e-05117    },118    "scheduler": "WarmupLinear",119    "steps_per_epoch": null,120    "warmup_steps": 10500,121    "weight_decay": 0.01122}123```124 125 126## Full Model Architecture127```128SentenceTransformer(129  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 130  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})131)132```133 134## Citing & Authors135 136Feel free to call my model CISR (Content Independent Style Representations) to save space in articles, as done by others.137 138```139@inproceedings{wegmann-etal-2022-author,140    title = "Same Author or Just Same Topic? Towards Content-Independent Style Representations",141    author = "Wegmann, Anna  and142      Schraagen, Marijn  and143      Nguyen, Dong",144    booktitle = "Proceedings of the 7th Workshop on Representation Learning for NLP",145    month = may,146    year = "2022",147    address = "Dublin, Ireland",148    publisher = "Association for Computational Linguistics",149    url = "https://aclanthology.org/2022.repl4nlp-1.26",150    pages = "249--268",151    abstract = "Linguistic style is an integral component of language. Recent advances in the development of style representations have increasingly used training objectives from authorship verification (AV){''}:'' Do two texts have the same author? The assumption underlying the AV training task (same author approximates same writing style) enables self-supervised and, thus, extensive training. However, a good performance on the AV task does not ensure good {``}general-purpose{''} style representations. For example, as the same author might typically write about certain topics, representations trained on AV might also encode content information instead of style alone. We introduce a variation of the AV training task that controls for content using conversation or domain labels. We evaluate whether known style dimensions are represented and preferred over content information through an original variation to the recently proposed STEL framework. We find that representations trained by controlling for conversation are better than representations trained with domain or no content control at representing style independent from content.",152}153```