quickmt/quickmt-train.de-en

1---2license: cc-by-4.03dataset_info:4  features:5  - name: de6    dtype: string7  - name: en8    dtype: string9  - name: sco10    dtype: float6411  splits:12  - name: train13    num_bytes: 12397269224214    num_examples: 52319910015  download_size: 8620425184016  dataset_size: 12397269224217configs:18- config_name: default19  data_files:20  - split: train21    path: data/train-*22task_categories:23- translation24language:25- de26- en27---28 29 30# `quickmt` de-en Training Corpus31 32Contains the following datasets downloaded with `mtdata` after deduplication and [basic filtering](https://github.com/quickmt/quickmt/blob/main/quickmt/scripts/clean.py) with `quickmt`:33 34* Statmt-commoncrawl_wmt13-1-deu-eng35* Statmt-europarl_wmt13-7-deu-eng36* Statmt-news_commentary_wmt18-13-deu-eng37* Statmt-europarl-9-deu-eng38* Statmt-europarl-7-deu-eng39* Statmt-news_commentary-14-deu-eng40* Statmt-news_commentary-15-deu-eng41* Statmt-news_commentary-16-deu-eng42* Statmt-news_commentary-17-deu-eng43* Statmt-news_commentary-18-deu-eng44* Statmt-news_commentary-18.1-deu-eng45* Statmt-newstest_deen-2014-deu-eng46* Statmt-newstest_ende-2015-eng-deu47* Statmt-newstest_deen-2015-deu-eng48* Statmt-newstest_deen-2016-deu-eng49* Statmt-newstest_ende-2016-eng-deu50* Statmt-newstest_ende-2017-eng-deu51* Statmt-newstest_deen-2017-deu-eng52* Statmt-newstest_ende-2018-eng-deu53* Statmt-newstest_deen-2018-deu-eng54* Statmt-newstest_deen-2019-deu-eng55* Statmt-newstest_ende-2019-eng-deu56* Statmt-newstest-2009-eng-deu57* Statmt-newstest-2010-eng-deu58* Statmt-newstest-2011-eng-deu59* Statmt-newstest-2012-eng-deu60* Statmt-newstest-2013-eng-deu61* Statmt-newstest_deen-2020-deu-eng62* Statmt-newstest_ende-2020-eng-deu63* Statmt-newstestb_deen-2020-deu-eng64* Statmt-newstestb_ende-2020-eng-deu65* Statmt-newstest_deen-2021-deu-eng66* Statmt-newstest_ende-2021-eng-deu67* Statmt-europarl-10-deu-eng68* Tilde-eesc-2017-deu-eng69* Tilde-ema-2016-deu-eng70* Tilde-airbaltic-1-deu-eng71* Tilde-czechtourism-1-deu-eng72* Tilde-ecb-2017-deu-eng73* Tilde-rapid-2016-deu-eng74* Tilde-rapid-2019-deu-eng75* Facebook-wikimatrix-1-deu-eng76* Lindat-khresmoi_summary_test-2-deu-eng77* Lindat-khresmoi_summary_dev-2-deu-eng78* Neulab-tedtalks_train-1-eng-deu79* Neulab-tedtalks_test-1-eng-deu80* Neulab-tedtalks_dev-1-eng-deu81* ELRC-swedish_labour_part2-1-deu-eng82* ELRC-rights_arrested-1-deu-eng83* ELRC-swedish_labour_part1-1-deu-eng84* ELRC-swedish_social_security-1-deu-eng85* ELRC-swedish_work_environment-1-deu-eng86* ELRC-federal_constitutional_law_austria-1-deu_DE-eng_GB87* ELRC-bmvi_publications-1-deu_DE-eng_GB88* ELRC-bmvi_website-1-deu_DE-eng_GB89* ELRC-bmi_brochure_civil_protection-1-deu_DE-eng_GB90* ELRC-bmi_brochures_2016-1-deu_DE-eng_GB91* ELRC-bmi_brochures_2011_2015-1-deu_DE-eng_GB92* ELRC-sip-1-deu_LU-eng93* ELRC-luxembourg.lu-1-deu-eng94* ELRC-federal_foreign_berlin-1-deu_DE-eng_GB95* ELRC-presidency-1-deu-eng96* ELRC-by_presidency_council_held_by_austria_2006-1-deu-eng97* ELRC-by_presidency_council_held_by_luxembourg_2015-1-deu-eng98* ELRC-charter_values_citizenship_integration-1-deu-eng99* ELRC-euipo_law-1-deu-eng100* ELRC-information_portal_german_state_chancellery-1-deu_DE-eng101* ELRC-german_foreign_office_2016-1-deu_DE-eng_GB102* ELRC-german_foreign_office_2017-1-deu_DE-eng_GB103* ELRC-german_foreign_office_2018-1-deu_DE-eng_GB104* ELRC-euipo_list-1-deu-eng105* ELRC-cordis_news-1-deu-eng106* ELRC-cordis_results_brief-1-deu-eng107* ELRC-energy_report_city_vienna-1-deu_AT-eng_GB108* ELRC-austrian_research_technology_report_2015-1-deu_AT-eng_GB109* ELRC-2017_activity_report_hohe_tauern_park-1-deu_AT-eng_GB110* ELRC-vienna_environmental_report2004_2005-1-deu_AT-eng_GB111* ELRC-euipo_2017-1-deu-eng112* ELRC-portal_oficial_turismo_españa_www.spain.info-1-deu_DE-eng_GB113* ELRC-artigos_visitportugal_2007-1-deu-eng114* ELRC-localidades_2007-1-deu-eng115* ELRC-museus_2007-1-deu-eng116* ELRC-arquitectura_2007-1-deu-eng117* ELRC-património_açores_2006-1-deu-eng118* ELRC-monumentos_2007-1-deu-eng119* ELRC-parques_e_reservas_2007-1-deu-eng120* ELRC-praias_2007-1-deu-eng121* ELRC-emea-1-deu-eng122* ELRC-vaccination-1-deu-eng123* ELRC-eu_publications_medical_v2-1-deu-eng124* ELRC-wikipedia_health-1-deu-eng125* ELRC-antibiotic-1-deu-eng126* ELRC-europarl_covid-1-deu-eng127* ELRC-ec_europa_covid-1-deu-eng128* ELRC-eur_lex_covid-1-deu-eng129* ELRC-presscorner_covid-1-deu-eng130* ELRC-development_funds_regional_policy-1-deu-eng131* ELRC-customer_support_mt_test_set-1-deu-eng132* ELRC-scipar-1-deu-eng133* ELRC-information_crime_victims_processed-1-eng-deu134* ELRC-web_acquired_data_related_to_scientific_research-1-eng-deu135* ELRC-hrw_dataset_v1-1-eng-deu136* ELRC-cef_data_marketplace-1-eng-deu137* EU-ecdc-1-eng-deu138* EU-eac_forms-1-deu-eng139* EU-eac_reference-1-deu-eng140* EU-dcep-1-deu-eng141* LinguaTools-wikititles-2014-deu-eng142* OPUS-books-v1-deu-eng143* OPUS-dgt-v2019-deu-eng144* OPUS-dgt-v4-deu-eng145* OPUS-ecb-v1-deu-eng146* OPUS-ecdc-v20160316-deu-eng147* OPUS-elitr_eca-v1-deu-eng148* OPUS-elra_w0143-v1-deu-eng149* OPUS-elra_w0197-v1-deu-eng_GB150* OPUS-elra_w0198-v1-deu-eng_GB151* OPUS-elra_w0199-v1-deu-eng_GB152* OPUS-elra_w0200-v1-deu-eng_GB153* OPUS-elra_w0201-v1-deu-eng154* OPUS-elra_w0301-v1-deu-eng155* OPUS-elrc_1077_euipo_law-v1-deu-eng156* OPUS-elrc_1086_information_portal_g-v1-deu-eng157* OPUS-elrc_1088_german_foreign_offic-v1-deu-eng158* OPUS-elrc_1089_german_foreign_offic-v1-deu-eng159* OPUS-elrc_1090_german_foreign_offic-v1-deu-eng160* OPUS-elrc_1092_euipo_list-v1-deu-eng161* OPUS-elrc_1117_cordis_news-v1-deu-eng162* OPUS-elrc_1121_cordis_results_brief-v1-deu-eng163* OPUS-elrc_1238_energy_report_city-v1-deu-eng164* OPUS-elrc_1240_austrian_research_te-v1-deu-eng165* OPUS-elrc_1241_2017_activity_report-v1-deu-eng166* OPUS-elrc_1243_vienna_environmental-v1-deu-eng167* OPUS-elrc_2014_euipo_2017-v1-deu-eng168* OPUS-elrc_2410_portal_oficial_turis-v1-deu-eng169* OPUS-elrc_2612_artigos_visitportuga-v1-deu-eng170* OPUS-elrc_2614_localidades_2007-v1-deu-eng171* OPUS-elrc_2616_museus_2007-v1-deu-eng172* OPUS-elrc_2622_arquitectura_2007-v1-deu-eng173* OPUS-elrc_2623_patrimnio_aores_2006-v1-deu-eng174* OPUS-elrc_2638_monumentos_2007-v1-deu-eng175* OPUS-elrc_2639_parques_e_reservas-v1-deu-eng176* OPUS-elrc_2641_praias_2007-v1-deu-eng177* OPUS-elrc_2714_emea-v1-deu-eng178* OPUS-elrc_2736_vaccination-v1-deu-eng179* OPUS-elrc_2875_eu_publications_medi-v1-deu-eng180* OPUS-elrc_3063_wikipedia_health-v1-deu-eng181* OPUS-elrc_3202_antibiotic-v1-deu-eng182* OPUS-elrc_3293_europarl_covid-v1-deu-eng183* OPUS-elrc_3464_ec_europa_covid-v1-deu-eng184* OPUS-elrc_3565_eur_lex_covid-v1-deu-eng185* OPUS-elrc_3606_presscorner_covid-v1-deu-eng186* OPUS-elrc_3852_development_funds_re-v1-deu-eng187* OPUS-elrc_401_swedish_labour_part2-v1-deu-eng188* OPUS-elrc_403_rights_arrested-v1-deu-eng189* OPUS-elrc_406_swedish_labour_part1-v1-deu-eng190* OPUS-elrc_416_swedish_social_secur-v1-deu-eng191* OPUS-elrc_417_swedish_work_environ-v1-deu-eng192* OPUS-elrc_4992_customer_support_mt-v1-deu-eng193* OPUS-elrc_5067_scipar-v1-deu-eng194* OPUS-elrc_5220_information_crime_vi-v1-deu-eng195* OPUS-elrc_621_federal_constitution-v1-deu-eng196* OPUS-elrc_630_bmvi_publications-v1-deu-eng197* OPUS-elrc_631_bmvi_website-v1-deu-eng198* OPUS-elrc_632_bmi_brochure_civil-v1-deu-eng199* OPUS-elrc_633_bmi_brochures_2016-v1-deu-eng200* OPUS-elrc_634_bmi_brochures_2011-v1-deu-eng201* OPUS-elrc_637_sip-v1-deu-eng202* OPUS-elrc_638_luxembourg.lu-v1-deu-eng203* OPUS-elrc_642_federal_foreign_berl-v1-deu-eng204* OPUS-elrc_774_presidency-v1-deu-eng205* OPUS-elrc_775_by_presidency_counci-v1-deu-eng206* OPUS-elrc_776_by_presidency_counci-v1-deu-eng207* OPUS-elrc_832_charter_values_citiz-v1-deu-eng208* OPUS-elrc_arquitectura_2007-v1-deu-eng209* OPUS-elrc_artigos_visitportuga-v1-deu-eng210* OPUS-elrc_cordis_news-v1-deu-eng211* OPUS-elrc_cordis_results-v1-deu-eng212* OPUS-elrc_ec_europa-v1-deu-eng213* OPUS-elrc_emea-v1-deu-eng214* OPUS-elrc_euipo_2017-v1-deu-eng215* OPUS-elrc_euipo_law-v1-deu-eng216* OPUS-elrc_euipo_list-v1-deu-eng217* OPUS-elrc_europarl_covid-v1-deu-eng218* OPUS-elrc_eur_lex-v1-deu-eng219* OPUS-elrc_eu_publications-v1-deu-eng220* OPUS-elrc_federal_foreign-v1-deu-eng_GB221* OPUS-elrc_german_foreign-v1-deu-eng_GB222* OPUS-elrc_information_portal-v1-deu-eng223* OPUS-elrc_localidades_2007-v1-deu-eng224* OPUS-elrc_museus_2007-v1-deu-eng225* OPUS-elrc_parques_e-v1-deu-eng226* OPUS-elrc_patrimnio_aores-v1-deu-eng227* OPUS-elrc_praias_2007-v1-deu-eng228* OPUS-elrc_swedish_labour-v1-deu-eng229* OPUS-elrc_termitur-v1-deu-eng230* OPUS-elrc_antibiotic-v1-deu-eng231* OPUS-elrc_presscorner_covid-v1-deu-eng232* OPUS-elrc_vaccination-v1-deu-eng233* OPUS-elrc_wikipedia_health-v1-deu-eng234* OPUS-elrc_2682-v1-deu-eng235* OPUS-elrc_2922-v1-deu-eng236* OPUS-elrc_2923-v1-deu-eng237* OPUS-elrc_3382-v1-deu-eng238* OPUS-emea-v3-deu-eng239* OPUS-eubookshop-v2-deu-eng240* OPUS-euconst-v1-deu-eng241* OPUS-europat-v1-deu-eng242* OPUS-europat-v2-deu-eng243* OPUS-europat-v3-deu-eng244* OPUS-europarl-v3-deu-eng245* OPUS-europarl-v7-deu-eng246* OPUS-europarl-v8-deu-eng247* OPUS-gnome-v1-deu-eng248* OPUS-globalvoices-v2015-deu-eng249* OPUS-globalvoices-v2017q3-deu-eng250* OPUS-globalvoices-v2018q4-deu-eng251* OPUS-jrc_acquis-v3.0-deu-eng252* OPUS-kde4-v2-deu-eng253* OPUS-kdedoc-v1-deu-eng_GB254* OPUS-mpc1-v1-deu-eng255* OPUS-multiun-v1-deu-eng256* OPUS-nllb-v1-deu-eng257* OPUS-neulab_tedtalks-v1-deu-eng258* OPUS-news_commentary-v11-deu-eng259* OPUS-news_commentary-v14-deu-eng260* OPUS-news_commentary-v16-deu-eng261* OPUS-news_commentary-v9.0-deu-eng262* OPUS-news_commentary-v9.1-deu-eng263* OPUS-openoffice-v2-deu-eng264* OPUS-openoffice-v3-deu-eng_GB265* OPUS-opensubtitles-v1-deu-eng266* OPUS-opensubtitles-v2016-deu-eng267* OPUS-opensubtitles-v2018-deu-eng268* OPUS-php-v1-deu-eng269* OPUS-paracrawl-v9-deu-eng270* OPUS-qed-v2.0a-deu-eng271* OPUS-rf-v1-deu-eng272* OPUS-salome-v1-deu-eng273* OPUS-stanfordnlp_nmt-v1.0-eng-deu274* OPUS-ted2013-v1.1-deu-eng275* OPUS-ted2020-v1-deu-eng276* OPUS-tanzil-v1-deu-eng277* OPUS-tatoeba-v2-deu-eng278* OPUS-tatoeba-v20190709-deu-eng279* OPUS-tatoeba-v20200531-deu-eng280* OPUS-tatoeba-v20201109-deu-eng281* OPUS-tatoeba-v20210310-deu-eng282* OPUS-tatoeba-v20210722-deu-eng283* OPUS-tatoeba-v20220303-deu-eng284* OPUS-tatoeba-v20230412-deu-eng285* OPUS-tildemodel-v2018-deu-eng286* OPUS-ubuntu-v14.10-deu-eng287* OPUS-wmt_news-v2014-deu-eng288* OPUS-wmt_news-v2019-deu-eng289* OPUS-wikimatrix-v1-deu-eng290* OPUS-wikipedia-v1.0-deu-eng291* OPUS-bible_uedin-v1-deu-eng292* OPUS-tldr_pages-v20230829-deu-eng