mteb/PawsXPairClassification

1---2annotations_creators:3- human-annotated4language:5- cmn6- deu7- eng8- fra9- jpn10- kor11- spa12license: other13multilinguality: translated14source_datasets:15- google-research-datasets/paws-x16task_categories:17- text-classification18task_ids:19- semantic-similarity-classification20- natural-language-inference21dataset_info:22- config_name: de23  features:24  - name: sentence125    dtype: string26  - name: sentence227    dtype: string28  - name: labels29    dtype: int6430  splits:31  - name: test32    num_bytes: 51620633    num_examples: 200034  - name: validation35    num_bytes: 50600136    num_examples: 200037  download_size: 67080138  dataset_size: 102220739- config_name: en40  features:41  - name: sentence142    dtype: string43  - name: sentence244    dtype: string45  - name: labels46    dtype: int6447  splits:48  - name: test49    num_bytes: 48672650    num_examples: 200051  - name: validation52    num_bytes: 48427953    num_examples: 200054  download_size: 59162455  dataset_size: 97100556- config_name: es57  features:58  - name: sentence159    dtype: string60  - name: sentence261    dtype: string62  - name: labels63    dtype: int6464  splits:65  - name: test66    num_bytes: 51110367    num_examples: 200068  - name: validation69    num_bytes: 50588070    num_examples: 200071  download_size: 66132372  dataset_size: 101698373- config_name: fr74  features:75  - name: sentence176    dtype: string77  - name: sentence278    dtype: string79  - name: labels80    dtype: int6481  splits:82  - name: test83    num_bytes: 52709384    num_examples: 200085  - name: validation86    num_bytes: 52502387    num_examples: 200088  download_size: 67358489  dataset_size: 105211690- config_name: ja91  features:92  - name: sentence193    dtype: string94  - name: sentence295    dtype: string96  - name: labels97    dtype: int6498  splits:99  - name: test100    num_bytes: 660628101    num_examples: 2000102  - name: validation103    num_bytes: 653770104    num_examples: 2000105  download_size: 784977106  dataset_size: 1314398107- config_name: ko108  features:109  - name: sentence1110    dtype: string111  - name: sentence2112    dtype: string113  - name: labels114    dtype: int64115  splits:116  - name: test117    num_bytes: 554292118    num_examples: 2000119  - name: validation120    num_bytes: 546867121    num_examples: 2000122  download_size: 709466123  dataset_size: 1101159124- config_name: zh125  features:126  - name: sentence1127    dtype: string128  - name: sentence2129    dtype: string130  - name: labels131    dtype: int64132  splits:133  - name: test134    num_bytes: 466636135    num_examples: 2000136  - name: validation137    num_bytes: 465110138    num_examples: 2000139  download_size: 666473140  dataset_size: 931746141configs:142- config_name: de143  data_files:144  - split: test145    path: de/test-*146  - split: validation147    path: de/validation-*148- config_name: en149  data_files:150  - split: test151    path: en/test-*152  - split: validation153    path: en/validation-*154- config_name: es155  data_files:156  - split: test157    path: es/test-*158  - split: validation159    path: es/validation-*160- config_name: fr161  data_files:162  - split: test163    path: fr/test-*164  - split: validation165    path: fr/validation-*166- config_name: ja167  data_files:168  - split: test169    path: ja/test-*170  - split: validation171    path: ja/validation-*172- config_name: ko173  data_files:174  - split: test175    path: ko/test-*176  - split: validation177    path: ko/validation-*178- config_name: zh179  data_files:180  - split: test181    path: zh/test-*182  - split: validation183    path: zh/validation-*184tags:185- mteb186- text187---188<!-- adapted from https://github.com/huggingface/huggingface_hub/blob/v0.30.2/src/huggingface_hub/templates/datasetcard_template.md -->189 190<div align="center" style="padding: 40px 20px; background-color: white; border-radius: 12px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05); max-width: 600px; margin: 0 auto;">191  <h1 style="font-size: 3.5rem; color: #1a1a1a; margin: 0 0 20px 0; letter-spacing: 2px; font-weight: 700;">PawsXPairClassification</h1>192  <div style="font-size: 1.5rem; color: #4a4a4a; margin-bottom: 5px; font-weight: 300;">An <a href="https://github.com/embeddings-benchmark/mteb" style="color: #2c5282; font-weight: 600; text-decoration: none;" onmouseover="this.style.textDecoration='underline'" onmouseout="this.style.textDecoration='none'">MTEB</a> dataset</div>193  <div style="font-size: 0.9rem; color: #2c5282; margin-top: 10px;">Massive Text Embedding Benchmark</div>194</div>195 196{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification197 198|               |                                             |199|---------------|---------------------------------------------|200| Task category | t2t                              |201| Domains       | Web, Encyclopaedic, Written                               |202| Reference     | https://arxiv.org/abs/1908.11828 |203 204 205 206 207## How to evaluate on this task208 209You can evaluate an embedding model on this dataset using the following code:210 211```python212import mteb213 214task = mteb.get_task("PawsXPairClassification")215evaluator = mteb.MTEB([task])216 217model = mteb.get_model(YOUR_MODEL)218evaluator.run(model)219```220 221<!-- Datasets want link to arxiv in readme to autolink dataset with paper -->222To learn more about how to run models on `mteb` task check out the [GitHub repository](https://github.com/embeddings-benchmark/mteb).223 224## Citation225 226If you use this dataset, please cite the dataset as well as [mteb](https://github.com/embeddings-benchmark/mteb), as this dataset likely includes additional processing as a part of the [MMTEB Contribution](https://github.com/embeddings-benchmark/mteb/tree/main/docs/mmteb).227 228```bibtex229 230@misc{yang2019pawsx,231  archiveprefix = {arXiv},232  author = {Yinfei Yang and Yuan Zhang and Chris Tar and Jason Baldridge},233  eprint = {1908.11828},234  primaryclass = {cs.CL},235  title = {PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification},236  year = {2019},237}238 239 240@article{enevoldsen2025mmtebmassivemultilingualtext,241  title={MMTEB: Massive Multilingual Text Embedding Benchmark},242  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},243  publisher = {arXiv},244  journal={arXiv preprint arXiv:2502.13595},245  year={2025},246  url={https://arxiv.org/abs/2502.13595},247  doi = {10.48550/arXiv.2502.13595},248}249 250@article{muennighoff2022mteb,251  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils},252  title = {MTEB: Massive Text Embedding Benchmark},253  publisher = {arXiv},254  journal={arXiv preprint arXiv:2210.07316},255  year = {2022}256  url = {https://arxiv.org/abs/2210.07316},257  doi = {10.48550/ARXIV.2210.07316},258}259```260 261# Dataset Statistics262<details>263  <summary> Dataset Statistics</summary>264 265The following code contains the descriptive statistics from the task. These can also be obtained using:266 267```python268import mteb269 270task = mteb.get_task("PawsXPairClassification")271 272desc_stats = task.metadata.descriptive_stats273```274 275```json276{}277```278 279</details>280 281---282*This dataset card was automatically generated using [MTEB](https://github.com/embeddings-benchmark/mteb)*