vector/test_demo

1---2annotaeators:3- found4language_creators:5- found6language:7- cn8 9---10 11### Dataset Summary 12Placeholder13You can load the dataset via:14```15import datasets16data = datasets.load_dataset('GEM/wiki_lingua')17```18The data loader can be found [here](https://huggingface.co/datasets/GEM/wiki_lingua).19#### website20None (See Repository)21#### paper22https://www.aclweb.org/anthology/2020.findings-emnlp.360/23#### authors24Faisal Ladhak (Columbia University), Esin Durmus (Stanford University), Claire Cardie (Cornell University), Kathleen McKeown (Columbia University)25## Dataset Overview26### Where to find the Data and its Documentation27#### Webpage28<!-- info: What is the webpage for the dataset (if it exists)? -->29<!-- scope: telescope -->30None (See Repository)31#### Download32<!-- info: What is the link to where the original dataset is hosted? -->33<!-- scope: telescope -->34https://github.com/esdurmus/Wikilingua35#### Paper36<!-- info: What is the link to the paper describing the dataset (open access preferred)? -->37<!-- scope: telescope -->38https://www.aclweb.org/anthology/2020.findings-emnlp.360/39#### BibTex40<!-- info: Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex. -->41<!-- scope: microscope -->42@inproceedings{ladhak-etal-2020-wikilingua,43    title = "{W}iki{L}ingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization",44    author = "Ladhak, Faisal  and45      Durmus, Esin  and46      Cardie, Claire  and47      McKeown, Kathleen",48    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",49    month = nov,50    year = "2020",51    address = "Online",52    publisher = "Association for Computational Linguistics",53    url = "https://aclanthology.org/2020.findings-emnlp.360",54    doi = "10.18653/v1/2020.findings-emnlp.360",55    pages = "4034--4048",56    abstract = "We introduce WikiLingua, a large-scale, multilingual dataset for the evaluation of cross-lingual abstractive summarization systems. We extract article and summary pairs in 18 languages from WikiHow, a high quality, collaborative resource of how-to guides on a diverse set of topics written by human authors. We create gold-standard article-summary alignments across languages by aligning the images that are used to describe each how-to step in an article. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset. We further propose a method for direct cross-lingual summarization (i.e., without requiring translation at inference time) by leveraging synthetic data and Neural Machine Translation as a pre-training step. Our method significantly outperforms the baseline approaches, while being more cost efficient during inference.",57}58#### Contact Name59<!-- quick -->60<!-- info: If known, provide the name of at least one person the reader can contact for questions about the dataset. -->61<!-- scope: periscope -->62Faisal Ladhak, Esin Durmus63#### Contact Email64<!-- info: If known, provide the email of at least one person the reader can contact for questions about the dataset. -->65<!-- scope: periscope -->66faisal@cs.columbia.edu, esdurmus@stanford.edu67#### Has a Leaderboard?68<!-- info: Does the dataset have an active leaderboard? -->69<!-- scope: telescope -->70no71### Languages and Intended Use72#### Multilingual?73<!-- quick -->74<!-- info: Is the dataset multilingual? -->75<!-- scope: telescope -->76yes77#### Covered Dialects78<!-- info: What dialects are covered? Are there multiple dialects per language? -->79<!-- scope: periscope -->80Dataset does not have multiple dialects per language.81#### Covered Languages82<!-- quick -->83<!-- info: What languages/dialects are covered in the dataset? -->84<!-- scope: telescope -->85`English`, `Spanish, Castilian`, `Portuguese`, `French`, `German`, `Russian`, `Italian`, `Indonesian`, `Dutch, Flemish`, `Arabic`, `Chinese`, `Vietnamese`, `Thai`, `Japanese`, `Korean`, `Hindi`, `Czech`, `Turkish`86#### Whose Language?87<!-- info: Whose language is in the dataset? -->88<!-- scope: periscope -->89No information about the user demographic is available.90#### License91<!-- quick -->92<!-- info: What is the license of the dataset? -->93<!-- scope: telescope -->94cc-by-3.0: Creative Commons Attribution 3.0 Unported95#### Intended Use96<!-- info: What is the intended use of the dataset? -->97<!-- scope: microscope -->98The dataset was intended to serve as a large-scale, high-quality benchmark dataset for cross-lingual summarization.99#### Primary Task100<!-- info: What primary task does the dataset support? -->101<!-- scope: telescope -->102Summarization103#### Communicative Goal104<!-- quick -->105<!-- info: Provide a short description of the communicative goal of a model trained for this task on this dataset. -->106<!-- scope: periscope -->107Produce a high quality summary for the given input article.108### Credit109#### Curation Organization Type(s)110<!-- info: In what kind of organization did the dataset curation happen? -->111<!-- scope: telescope -->112`academic`113#### Curation Organization(s)114<!-- info: Name the organization(s). -->115<!-- scope: periscope -->116Columbia University117#### Dataset Creators118<!-- info: Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s). -->119<!-- scope: microscope -->120Faisal Ladhak (Columbia University), Esin Durmus (Stanford University), Claire Cardie (Cornell University), Kathleen McKeown (Columbia University)121#### Who added the Dataset to GEM?122<!-- info: Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM. -->123<!-- scope: microscope -->124Jenny Chim (Queen Mary University of London), Faisal Ladhak (Columbia University)125### Dataset Structure126#### Data Fields127<!-- info: List and describe the fields present in the dataset. -->128<!-- scope: telescope -->129gem_id -- The id for the data instance.130source_language -- The language of the source article.131target_language -- The language of the target summary.132source -- The source document.133 134 135#### Example Instance136 137<!-- info: Provide a JSON formatted example of a typical instance in the dataset. -->138<!-- scope: periscope -->139{140    "gem_id": "wikilingua_crosslingual-train-12345",141    "gem_parent_id": "wikilingua_crosslingual-train-12345",142    "source_language": "fr",143    "target_language": "de",144    "source": "Document in fr",145    "target": "Summary in de",146}147#### Data Splits148 149<!-- info: Describe and name the splits in the dataset if there are more than one. -->150<!-- scope: periscope -->151The data is split into train/dev/test. In addition to the full test set, there's also a sampled version of the test set. 152 153#### Splitting Criteria154 155<!-- info: Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g., if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here. -->156<!-- scope: microscope -->157The data was split to ensure the same document would appear in the same split across languages so as to ensure there's no leakage into the test set.158 159 160 161## Dataset in GEM162 163### Rationale for Inclusion in GEM164 165#### Why is the Dataset in GEM?166 167<!-- info: What does this dataset contribute toward better generation evaluation and why is it part of GEM? -->168<!-- scope: microscope -->169This dataset provides a large-scale, high-quality resource for cross-lingual summarization in 18 languages, increasing the coverage of languages for the GEM summarization task. 170 171#### Similar Datasets172 173<!-- info: Do other datasets for the high level task exist? -->174<!-- scope: telescope -->175yes176 177#### Unique Language Coverage178 179<!-- info: Does this dataset cover other languages than other datasets for the same task? -->180<!-- scope: periscope -->181yes182 183#### Difference from other GEM datasets184 185<!-- info: What else sets this dataset apart from other similar datasets in GEM? -->186<!-- scope: microscope -->187XSum covers English news articles, and MLSum covers news articles in German and Spanish. 188In contrast, this dataset has how-to articles in 18 languages, substantially increasing the languages covered. Moreover, it also provides a a different domain than the other two datasets.189 190#### Ability that the Dataset measures191 192<!-- info: What aspect of model ability can be measured with this dataset? -->193<!-- scope: periscope -->194The ability to generate quality summaries across multiple languages.195 196 197### GEM-Specific Curation198 199#### Modificatied for GEM?200 201<!-- info: Has the GEM version of the dataset been modified in any way (data, processing, splits) from the original curated data? -->202<!-- scope: telescope -->203yes204 205#### GEM Modifications206 207<!-- info: What changes have been made to he original dataset? -->208<!-- scope: periscope -->209`other`210 211#### Modification Details212 213<!-- info: For each of these changes, described them in more details and provided the intended purpose of the modification -->214<!-- scope: microscope -->215Previous version had separate data loaders for each language. In this version, we've created a single monolingual data loader, which contains monolingual data in each of the 18 languages. In addition, we've also created a single cross-lingual data loader across all the language pairs in the dataset.  216 217#### Additional Splits?218 219<!-- info: Does GEM provide additional splits to the dataset? -->220<!-- scope: telescope -->221no222 223 224### Getting Started with the Task225 226 227 228 229## Previous Results230 231### Previous Results232 233#### Measured Model Abilities234 235<!-- info: What aspect of model ability can be measured with this dataset? -->236<!-- scope: telescope -->237Ability to summarize content across different languages.238 239#### Metrics240 241<!-- info: What metrics are typically used for this task? -->242<!-- scope: periscope -->243`ROUGE`244 245#### Proposed Evaluation246 247<!-- info: List and describe the purpose of the metrics and evaluation methodology (including human evaluation) that the dataset creators used when introducing this task. -->248<!-- scope: microscope -->249ROUGE is used to measure content selection by comparing word overlap with reference summaries. In addition, the authors of the dataset also used human evaluation to evaluate content selection and fluency of the systems.250 251#### Previous results available?252 253<!-- info: Are previous results available? -->254<!-- scope: telescope -->255no256 257 258 259## Dataset Curation260 261### Original Curation262 263#### Original Curation Rationale264 265<!-- info: Original curation rationale -->266<!-- scope: telescope -->267The dataset was created in order to enable new approaches for cross-lingual and multilingual summarization, which are currently understudied as well as open up inetersting new directions for research in summarization. E.g., exploration of multi-source cross-lingual architectures, i.e. models that can summarize from multiple source languages into a target language, building models that can summarize articles from any language to any other language for a given set of languages.268 269#### Communicative Goal270 271<!-- info: What was the communicative goal? -->272<!-- scope: periscope -->273Given an input article, produce a high quality summary of the article in the target language.274 275#### Sourced from Different Sources276 277<!-- info: Is the dataset aggregated from different data sources? -->278<!-- scope: telescope -->279no280 281 282### Language Data283 284#### How was Language Data Obtained?285 286<!-- info: How was the language data obtained? -->287<!-- scope: telescope -->288`Found`289 290#### Where was it found?291 292<!-- info: If found, where from? -->293<!-- scope: telescope -->294`Single website`295 296#### Language Producers297 298<!-- info: What further information do we have on the language producers? -->299<!-- scope: microscope -->300WikiHow, which is an online resource of how-to guides (written and reviewed by human authors) is used as the data source. 301 302#### Topics Covered303 304<!-- info: Does the language in the dataset focus on specific topics? How would you describe them? -->305<!-- scope: periscope -->306The articles cover 19 broad categories including health, arts and entertainment, personal care and style, travel, education and communications, etc. The categories cover a broad set of genres and topics.307 308#### Data Validation309 310<!-- info: Was the text validated by a different worker or a data curator? -->311<!-- scope: telescope -->312not validated313 314#### Was Data Filtered?315 316<!-- info: Were text instances selected or filtered? -->317<!-- scope: telescope -->318not filtered319 320 321### Structured Annotations322 323#### Additional Annotations?324 325<!-- quick -->326<!-- info: Does the dataset have additional annotations for each instance? -->327<!-- scope: telescope -->328none329 330#### Annotation Service?331 332<!-- info: Was an annotation service used? -->333<!-- scope: telescope -->334no335 336 337### Consent338 339#### Any Consent Policy?340 341<!-- info: Was there a consent policy involved when gathering the data? -->342<!-- scope: telescope -->343yes344 345#### Consent Policy Details346 347<!-- info: What was the consent policy? -->348<!-- scope: microscope -->349(1) Text Content. All text posted by Users to the Service is sub-licensed by wikiHow to other Users under a Creative Commons license as provided herein. The Creative Commons license allows such text content be used freely for non-commercial purposes, so long as it is used and attributed to the original author as specified under the terms of the license. Allowing free republication of our articles helps wikiHow achieve its mission by providing instruction on solving the problems of everyday life to more people for free. In order to support this goal, wikiHow hereby grants each User of the Service a license to all text content that Users contribute to the Service under the terms and conditions of a Creative Commons CC BY-NC-SA 3.0 License. Please be sure to read the terms of the license carefully. You continue to own all right, title, and interest in and to your User Content, and you are free to distribute it as you wish, whether for commercial or non-commercial purposes.350 351#### Other Consented Downstream Use352 353<!-- info: What other downstream uses of the data did the original data creators and the data curators consent to? -->354<!-- scope: microscope -->355The data is made freely available under the Creative Commons license, therefore there are no restrictions about downstream uses as long is it's for non-commercial purposes.356 357 358### Private Identifying Information (PII)359 360#### Contains PII?361 362<!-- quick -->363<!-- info: Does the source language data likely contain Personal Identifying Information about the data creators or subjects? -->364<!-- scope: telescope -->365no PII366 367#### Justification for no PII368 369<!-- info: Provide a justification for selecting `no PII` above. -->370<!-- scope: periscope -->371Only the article text and summaries were collected. No user information was retained in the dataset.372 373 374### Maintenance375 376#### Any Maintenance Plan?377 378<!-- info: Does the original dataset have a maintenance plan? -->379<!-- scope: telescope -->380no381 382 383 384## Broader Social Context385 386### Previous Work on the Social Impact of the Dataset387 388#### Usage of Models based on the Data389 390<!-- info: Are you aware of cases where models trained on the task featured in this dataset ore related tasks have been used in automated systems? -->391<!-- scope: telescope -->392yes - other datasets featuring the same task393 394 395### Impact on Under-Served Communities396 397#### Addresses needs of underserved Communities?398 399<!-- info: Does this dataset address the needs of communities that are traditionally underserved in language technology, and particularly language generation technology? Communities may be underserved for exemple because their language, language variety, or social or geographical context is underepresented in NLP and NLG resources (datasets and models). -->400<!-- scope: telescope -->401no402 403 404### Discussion of Biases405 406#### Any Documented Social Biases?407 408<!-- info: Are there documented social biases in the dataset? Biases in this context are variations in the ways members of different social categories are represented that can have harmful downstream consequences for members of the more disadvantaged group. -->409<!-- scope: telescope -->410yes411 412 413 414## Considerations for Using the Data415 416### PII Risks and Liability417 418 419 420### Licenses421 422#### Copyright Restrictions on the Dataset423 424<!-- info: Based on your answers in the Intended Use part of the Data Overview Section, which of the following best describe the copyright and licensing status of the dataset? -->425<!-- scope: periscope -->426`non-commercial use only`427 428#### Copyright Restrictions on the Language Data429 430<!-- info: Based on your answers in the Language part of the Data Curation Section, which of the following best describe the copyright and licensing status of the underlying language data? -->431<!-- scope: periscope -->432`non-commercial use only`433 434 435### Known Technical Limitations436 437 438 439 440