Lajavaness/CrossEncoder-camembert-large

1---2pipeline_tag: text-ranking3language: fr4datasets:5- stsb_multi_mt6tags:7- Text8- Sentence Similarity9- Sentence-Embedding10- camembert-base11license: apache-2.012model-index:13- name: CrossEncoder-camembert-large by Van Tuan DANG14  results:15  - task:16      type: Text Similarity17      name: Sentence-Embedding18    dataset:19      name: Text Similarity fr20      type: stsb_multi_mt21      args: fr22    metrics:23    - type: Pearson_correlation_coefficient24      value: 90.3425      name: Test Pearson correlation coefficient26---27 28## Model29 30Cross-Encoder Model for sentence-similarity31 32This model was is an improvement over the [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) offering greater robustness and better performance33 34## Training Data35This model was trained on the [STS benchmark dataset](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) and has been combined with [Augmented SBERT](https://aclanthology.org/2021.naacl-main.28.pdf). The model benefits from Pair Sampling Strategies using two models: [CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) and [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.36 37 38## Usage (Sentence-Transformers)39 40Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:41 42```43pip install -U sentence-transformers44```45 46Then you can use the model like this:47 48```python49from sentence_transformers import CrossEncoder50model = CrossEncoder('Lajavaness/CrossEncoder-camembert-large', max_length=512)51scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])52 53```54## Evaluation55The model can be evaluated as follows on the French test data of stsb.56```python57from sentence_transformers.readers import InputExample58from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator59from datasets import load_dataset60def convert_dataset(dataset):61    dataset_samples=[]62    for df in dataset:63        score = float(df['similarity_score'])/5.0  # Normalize score to range 0 ... 164        inp_example = InputExample(texts=[df['sentence1'], 65                                    df['sentence2']], label=score)66        dataset_samples.append(inp_example)67    return dataset_samples68 69# Loading the dataset for evaluation70df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")71df_test = load_dataset("stsb_multi_mt", name="fr", split="test")72 73# Convert the dataset for evaluation74 75# For Dev set:76dev_samples = convert_dataset(df_dev)77val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')78val_evaluator(model, output_path="./")79 80# For Test set, the Pearson and Spearman correlation are evaluated on many different benchmark datasets:81 82test_samples = convert_dataset(df_test)83test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')84test_evaluator(models, output_path="./")85```86**Test Result**: 87The performance is measured using Pearson and Spearman correlation:88 89- On dev90  91| Model  | Pearson correlation | Spearman correlation  |  #params  |92| ------------- | ------------- | ------------- |------------- |93| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.34 |90.15 | 336M |94| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.11 |90.01 | 336M |95 96- On test:97 98  99**Pearson score**100 101| Model                                  | STS-B  | STS12-fr | STS13-fr | STS14-fr | STS15-fr | STS16-fr | SICK-fr |102|---------------------------------------|--------|----------|----------|----------|----------|----------|---------|103| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)       | 88.63 | 90.76   | 88.24   | 90.22   | 92.23   | 82.31   | 84.61  | 104| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)      | 88.16 | 90.12   | 88.36   | 89.86   | 92.04   | 82.01   | 84.23  | 105 106**Spearman score**107 108 109| Model                                  | STS-B  | STS12-fr | STS13-fr | STS14-fr | STS15-fr | STS16-fr | SICK-fr |110|---------------------------------------|--------|----------|----------|----------|----------|----------|---------|111| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)      | 88.03 | 84.87   | 87.88   | 89.10   | 92.16   | 82.50   | 80.78  |112| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)     | 87.57 | 84.24   | 88.01   | 88.62   | 91.99   | 82.16   | 80.38  |113