Tomlim/myt5-large

MyT5

Model Details

MyT5 (Myte T5) is a multilingual language model based on T5 architecture. The model uses a morphologically-driven byte (MYTE) representation described in our paper Limisiewicz et al., 2024.

Model Description

<!-- Provide a longer summary of what this model is. -->

• —Developed by: Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, Luke Zettlemoyer
• —Funded by: University of Washington Fellowship, Charles University Grant Agency
• —Model type: T5
• —Language(s) (NLP): Multilingual
• —License: MIT

Model Sizes

• —[Small](https://huggingface.co/Tomlim/myt5-small): 300M parameters
• —[Base](https://huggingface.co/Tomlim/myt5-base): 582M parameters
• —[Large](https://huggingface.co/Tomlim/myt5-large): 1.2B parameters

Model Sources

<!-- Provide the basic links for the model. -->

• —[Repository](https://github.com/tomlimi/MYTE)
• —[Paper](https://arxiv.org/pdf/2403.10691.pdf)

How to Get Started with the Model

The snippet below shows the basic usage of the model for multilingual language modeling. Custom Tokenizer is available in GitHubrepository, in src/myt5/myt5_tokenizer.py. We also plan to release it on HuggingFace in the future.

from transformers import T5ForConditionalGeneration
from src.myt5.myt5_tokenizer import MyT5Tokenizer
import torch

MODEL_SIZE = "large" # small, base, or large

model = T5ForConditionalGeneration.from_pretrained(f"Tomlim/MyT5_{MODEL_SIZE}", use_safetensors=True)
tokenizer = MyT5Tokenizer()

pre_texts = ['"We now have',
            '„Mamy teraz myszy w wieku',
            '"""எங்களிடம் இப்போது']
post_texts = ['4-month-old mice that are non-diabetic that used to be diabetic," he added.',
              '4 miesięcy, które miały cukrzycę, ale zostały z niej wyleczone” – dodał.',
              '4-மாத-வயதுடைய எலி ஒன்று உள்ளது, முன்னர் அதற்கு நீரிழிவு இருந்தது தற்போது இல்லை"" என்று அவர் மேலும் கூறினார்."']

inputs = tokenizer(pre_texts, padding="longest", return_tensors="pt")
targets = tokenizer(post_texts, padding="longest", return_tensors="pt")


outputs = model(**inputs, labels=targets.input_ids)
probs = torch.nn.functional.softmax(outputs.logits, dim=-1)

Training Details

Training Data

The model was trained on the standard T5 task of restoring corrupted spans in the multilingual MC4 dataset.

Preprocessing

Instead of UTF-8 bytes, we used morphologically-driven byte representation. See the description in our paper for more details.

Training Hyperparameters

We used the same hyperparameters as in the original ByT5 paper. The only difference is that we decreased the number of training steps to 250,000 to avoid overfiting.

Computational Infrastructure

Models were trained on TPUs available through TPU Research Cloud (TRC). We used v3-8 TPU for training small and base models and v3-32 for a large model. The training for each instance took:

• —Small: 90h
• —Base: 230h
• —Large: 190h

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

MyT5 models are compared with reimplementation of ByT5 models trained for 250,000 steps.

Language Modeling

We have evaluated LM performance on multi-parallel FLORES 200 corpus. To compare the scores across languages and models, we used a normalized metric, i.e., Bit-per-English-Byte (BPEB).

Results

Byte-per-English-Bits and Inference times (average per Flores 200 sentence) averaged for three language groupings. The inference was run on an A40 GPU core.

Downstream Tasks

We tested the large model in four end-tasks: question answering, NER, semantic parsing, and machine translation. The test data come from XTREME-UP benchmark (Ruder, Clark et al., 2023), which covers mainly low-resource languages

Fine-tuning

In each task, we fine-tuned for all languages jointly. We used 1e-3 learning rate with square root decay and dropout of 0.1. The batch size and training varied across tasks:

• —NER: 128 examples per batch, 6000 steps
• —QA: 64 examples per batch, 6500 steps
• —Semantic Parsing: 64 examples per batch, 1000 steps
• —MT: 64 examples per batch, 10000 steps

Results

Inference Times per example (ms) ByT5 | 36.2 | 13.8 | 13.2 | 15.9 MyT5 | 35.6 | 12.6 | 12.4 | 12.6

The average result of XTREME-UP tasks across low-resource languages. The baseline results of mT5 and Flan-PaLM (in-context-learning evaluation) are reported in Ruder, Clark et al., 2023. The reported inference time is an average across evaluation examples; the inference was run on an A40 GPU core.

Citation

@misc{limisiewicz2024myte,
      title={MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling}, 
      author={Tomasz Limisiewicz and Terra Blevins and Hila Gonen and Orevaoghene Ahia and Luke Zettlemoyer},
      year={2024},
      eprint={2403.10691},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Model Card Author

Tomasz Limisiewicz