CoolFace
Modelpublic

Helsinki-NLP/opus-mt-tc-bible-big-deu_eng_fra_por_spa-mul

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
1likes192downloads
Model Card

opus-mt-tc-bible-big-deuengfraporspa-mul

Table of Contents

Model Details

Neural machine translation model for translating from unknown (deu+eng+fra+por+spa) to Multiple languages (mul). Note that many of the listed languages will not be well supported by the model as the training data is very limited for the majority of the languages. Translation performance varies a lot and for a large number of language pairs it will not work at all.

This model is part of the OPUS-MT project, an effort to make neural machine translation models widely available and accessible for many languages in the world. All models are originally trained using the amazing framework of Marian NMT, an efficient NMT implementation written in pure C++. The models have been converted to pyTorch using the transformers library by huggingface. Training data is taken from OPUS and training pipelines use the procedures of OPUS-MT-train. Model Description:

  • —Developed by: Language Technology Research Group at the University of Helsinki
  • —Model Type: Translation (transformer-big)
  • —Release: 2024-05-30
  • —License: Apache-2.0
  • —Language(s):
  • —Source Language(s): deu eng fra por spa
  • —Target Language(s): aai aar aau abi abk acd ace acf ach acm acn acr ade adj ady aeu aey afb afhLatn afr agd agn agu ahk aia aka akh aklLatn akp alj aln alp alq alt alz ame amh ami amiLatn amk amu amuLatn angLatn ann anp anv aoz apc apr apu ara arc arg arq arz asm aso ast atg atj atq aui auy ava avkLatn avn avu awa awb awx azeCyrl azeLatn azg azz azzLatn bak bal balLatn bam bamLatn ban bar bas bav bba bbo bbr bcl bcw bef beh bel bem ben bep bex bfa bfd bfo bgr bhl bho bhz bib bik bim bis biv bjr bjv bku bkv blh blt blz bmh bmk bmq bmu bmv bnp bod boj bomLatn bosCyrl bosLatn bov box bpr bps bpy bqc bqj bqp bre bru brx bss btd bth bto bts btt btx bua bud bug buk bul bus bvyLatn bwq bwu byn bzd bzh bzj bztLatn caa cab cac cak cakLatn cat cay cbkLatn cce cco ceb ces cfm cgc cha che chf chm chq chqLatn chr chu chv chy chz cjk cjkLatn cjo cjp cjpLatn cjv cjyHans cjyHant ckb cko cle cme cmn cmnHans cmnHant cmo cmr cnh cnhLatn cni cniLatn cnl cnr cnrLatn cnt cnw cok cop copCopt cor cos cot cpa cpu cre creLatn crh crn crs crx csb csbLatn csk cso csy cta ctd ctp ctu cuc cui cuk cut cux cwe cwt cya cym czt daa dad dagLatn dah dan ded deu dga dgi dig dik din diq div dje djk djkLatn dng dni dnj dob dop dopLatn drtLatn dsb dsh dtp dty dug dwsLatn dww dyi dyo dyu dzo efi egl ell emi eng enmLatn epo ess est eus ewe ext fai fal fao far fas fij fil fin fkvLatn fon for fra frd frmLatn froLatn frp frr fry fuc ful fur gag gah gaw gbm gcf gcfLatn gde gej gfk ghs gil gkn gla gle glg glk glv gnd gng gog gor gos got gotGoth gqr grc grcGrek grn gsw guc gud guh guj guo gur guw guwLatn gux gvf gvl gwi gwr gym gyr hag hat hau hauLatn haw hay hbo hboHebr hbs hbsCyrl hbsLatn hch heb heh her hif hifLatn hig hil hin hinLatn hla hlt hmn hne hnj hnn hns hoc hocWara hot hrv hrxLatn hsb hsn hui hun hus husLatn hvn hwc hye hyw hywArmn hywLatn iba ibo icr idoLatn ifa ifb ife ifk ifu ify ign igsLatn iii ikeLatn iku ikuLatn ileLatn ilo imo inaLatn ind inh inhLatn ino iou ipi ipk iri irk iry isl ita itv ium ixl ixlLatn izh izr jaa jaaBopo jaaHira jaaKana jaaYiii jac jakLatn jam jav javJava jbo jboCyrl jboLatn jbu jdtCyrl jmc jpaHebr jpn jun jvn kaa kab kac kal kam kan kao kasArab kasDeva kat kau kaz kazCyrl kbd kbm kbp kbpCans kbpEthi kbpGeor kbpGrek kbpHang kbpLatn kbpMlym kbpYiii kdc kdj kdl kdn kea kek kekLatn ken keo ker keu kew kez kgf kgk kha khm khz kia kik kin kirCyrl kjb kje kjh kjs kki kkj kle kma kmb kmg kmh kmo kmr kmu knc kne knj knk kno kog koi kok kom kon kpf kpg kpr kpv kpw kpz kqe kqf kqp kqw krc kri krj krl kru ksb ksh ksr ktb ktj kua kub kud kue kum kurArab kurCyrl kurLatn kus kvn kwf kxc kxm kyc kyf kyg kyq kzf laaLatn lac lad ladLatn lah lao las lat latLatn lav law lbe lcm ldnLatn lee lef lem leu lew lex lez lfnCyrl lfnLatn lgg lhu lia lid lif lij lim lin lip lit livLatn ljp lkt lldLatn lln lme lmo lnd lob lok lon louLatn lrc lsi ltz lua luc lug luo lus lutLatn luy lzzLatn maa mad mag mah mai maj mak mal mam mamLatn maq mar mau maw maxLatn maz mbb mbf mbt mcb mcp mcu mda mdf med mee mehLatn mek men meq mfe mfh mfi mfk mfq mfy mgd mgmLatn mgo mhi mhl mhx mhy mib mic mie mif mig mih mil mio mit mix mixLatn miy miz mjc mkd mks mlg mlh mlp mlt mmo mmx mna mnb mnf mnh mni mnrLatn mnw moa mog moh mol mon mop mor mos mox mpg mpm mpt mpx mqb mqj mri mrj mrw msa msaArab msaLatn msm mta muh mux muy mva mvp mvvLatn mwc mwl mwm mwv mww mxb mxt mya myb myk myu myv myw myx mzk mzm mzn mzw mzz naf nak nap nas nau nav nbl nca nch ncj ncl ncu nde ndo nds ndz neb nep new nfr ngtLatn ngu nguLatn nhe nhg nhgLatn nhi nhnLatn nhu nhw nhx nhy nia nif nii nij nim nin niu njm nlc nld nlvLatn nmz nnb nnbLatn nnh nno nnw nob nog non nop nor not nou novLatn npi npl npy nqo nsn nso nss nstLatn nsu ntm ntp ntr nuj nus nuy nwb nwi nya nyf nyn nyo nyy nzi oarHebr oarSyrc obo oci ofsLatn ojiLatn oku okv old omw ood oodLatn opm ori orm orvCyrl ospLatn oss otaArab otaLatn otaRohg otaSyrc otaThaa otaYezi ote otkOrkh otm otn otq ozm pab pad pag paiLatn pal pam pan panGuru pao pap pau pbi pbl pcd pckLatn pcm pdc pes pfl phnPhnx pib pih pihLatn pio pis pkb pli pls plt plw pmf pms pmyLatn pne pntGrek poe poh pol por pot potLatn ppk ppkLatn pplLatn prf prgLatn prs ptp ptu pus pwg pww quc qya qyaLatn rai rap rav rej rhgLatn rifLatn rim rmy roh rom ron rop rro rue rug run rup rus rwo sab sag sah san sanDeva sas sat satLatn sba sbd sbl scn sco sda sdh seh ses sgb sgs sgw sgz shi shiLatn shk shn shsLatn shyLatn sig sil sin sjnLatn skr sld slk sll slv sma sme smk sml smlLatn smn smo sna snc sndArab snp snw som sot soy spa spl spp sps sqi srd srm srn srpCyrl srq ssd ssw ssx stn stp stq sue suk sun sur sus suz swa swc swe swg swh swp sxb sxn syc sylSylo syr szb szl tab tac tah taj tam taq tat tbc tbl tbo tbz tcs tcy tel tem teo ter tet tfr tgk tgkCyrl tgkLatn tgl tglLatn tglTglg tgo tgp tha thk thv tig tik tim tir tkl tlb tlf tlh tlhLatn tlj tlx tlyLatn tmc tmh tmrHebr tmwLatn toh toi toiLatn toj ton tpa tpi tpm tpwLatn tpz trc trn trq trs trsLatn trv tsn tso tsw ttc tte ttr tts tuc tuf tuk tukLatn tum tur tvl twb twi twu txa tyjLatn tyv tzh tzj tzl tzlLatn tzmLatn tzmTfng tzo ubr ubu udm udu uig uigArab uigCyrl uigLatn ukr umb urd usa usp uspLatn uvl uzbCyrl uzbLatn vag vec ven vep vie viv vls vmw vmy volLatn vot votLatn vro vun wae waj wal wap war wbm wbp wed wln wmt wmw wnc wnu wob wol wsk wuu wuv xal xclArmn xclLatn xed xho xmf xog xon xrb xsb xsi xsm xsr xtd xtm xuo yal yam yaq yaz yby ycl ycn yid yli yml yon yor yua yueHans yueHant yut yuw zam zap zea zgh zha zia zlmArab zlmLatn zom zsmArab zsm_Latn zul zyp zza
  • —Original Model: opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip
  • —Resources for more information:
  • —OPUS-MT dashboard
  • —OPUS-MT-train GitHub Repo
  • —More information about MarianNMT models in the transformers library
  • —Tatoeba Translation Challenge
  • —HPLT bilingual data v1 (as part of the Tatoeba Translation Challenge dataset)
  • —A massively parallel Bible corpus

This is a multilingual translation model with multiple target languages. A sentence initial language token is required in the form of >>id<< (id = valid target language ID), e.g. >>aai<<

Uses

This model can be used for translation and text-to-text generation.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware that the model is trained on various public data sets that may contain content that is disturbing, offensive, and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).

Also note that many of the listed languages will not be well supported by the model as the training data is very limited for the majority of the languages. Translation performance varies a lot and for a large number of language pairs it will not work at all.

How to Get Started With the Model

A short example code:

python
from transformers import MarianMTModel, MarianTokenizer

src_text = [
    ">>aai<< Replace this with text in an accepted source language.",
    ">>zza<< This is the second sentence."
]

model_name = "pytorch-models/opus-mt-tc-bible-big-deu_eng_fra_por_spa-mul"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))

for t in translated:
    print( tokenizer.decode(t, skip_special_tokens=True) )

You can also use OPUS-MT models with the transformers pipelines, for example:

python
from transformers import pipeline
pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-bible-big-deu_eng_fra_por_spa-mul")
print(pipe(">>aai<< Replace this with text in an accepted source language."))

Training

Evaluation

langpairtestsetchr-FBLEU#sent#words
multi-multitatoeba-test-v2020-07-28-v2023-09-260.5502429.21000075838

Citation Information

bibtex
@article{tiedemann2023democratizing,
  title={Democratizing neural machine translation with {OPUS-MT}},
  author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
  journal={Language Resources and Evaluation},
  number={58},
  pages={713--755},
  year={2023},
  publisher={Springer Nature},
  issn={1574-0218},
  doi={10.1007/s10579-023-09704-w}
}

@inproceedings{tiedemann-thottingal-2020-opus,
    title = "{OPUS}-{MT} {--} Building open translation services for the World",
    author = {Tiedemann, J{\"o}rg  and Thottingal, Santhosh},
    booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
    month = nov,
    year = "2020",
    address = "Lisboa, Portugal",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2020.eamt-1.61",
    pages = "479--480",
}

@inproceedings{tiedemann-2020-tatoeba,
    title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
    author = {Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the Fifth Conference on Machine Translation",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.wmt-1.139",
    pages = "1174--1182",
}

Acknowledgements

The work is supported by the HPLT project, funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350. We are also grateful for the generous computational resources and IT infrastructure provided by CSC -- IT Center for Science, Finland, and the EuroHPC supercomputer LUMI.

Model conversion info

  • —transformers version: 4.45.1
  • —OPUS-MT git hash: 0882077
  • —port time: Wed Oct 9 18:54:16 EEST 2024
  • —port machine: LM0-400-22516.local