hrezaei/T5LA

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

T5LA

This model is part of the work published in the paper Interactive Text Games: Lookahead Is All You Need!

Four models are introduced in the above paper:

• —nanoGPTLA
• —nanoGPTLAA
• —nanoGPTLAA2
• —nanoGPTLAE

These models are implemented in this repository which is a customized version of nanoGPT.

The same variations are also implemented in this fork of Transformers library, on top of Google-t5/T5 implementation. These models are also trained and published as follows:

• —T5LA
• —T5LAA
• —T5LAA2
• —T5LAE

All the above models are on the scale of GPT2 (~100M parameters). The work is in progress to train them on larger scales.

Model description

This model is not fine-tuned on any instruction or human feedback datasets. It is just pre-trained on the HuggingFaceFW/fineweb sample-10BT dataset. It achieves the following results on the evaluation set:

• —Loss: 5.5467
• —Accuracy: 0.0322

Since the above fork is not merged into the main Transformers library yet, if you need to load it with AutoModel.frompretrained(), you need to first install Transformers from [this branch](https://github.com/HRezaei/transformers/tree/feature/lookaheadmodels), which contains the code for T5LA models. This can be done by:

pip install git+https://github.com/HRezaei/transformers.git@feature/lookahead_models

Intended uses & limitations

The model is designed to predict not only the next immediate token after the prompt (which normal LLMs do), but also to predict the second, third, ..., up to K next tokens, conditioned on the prompt. These future predictions can be useful for approximated ranking, where a set of potential responses are needed to be ranked based on the approximated probability of their tokens conditioned on the prompt, rather than conditioned on their previous tokens.

The main limitation is that future predictions are generaly not suitable for generating text, as they don't consider token interdependencies, i.e. the future tokens are not conditioned on the previous tokens. Thus, for generation, one should rely only on the next immediate token. However, the quality of next immediate token prediction is also degraded, because during training, the loss function has more terms to minimize (one term for next immediate token like original LLMs, and one extra term per each future tokens).

Training and evaluation data

This model is not fine-tuned on any instruction or human feedback datasets. It is just pre-trained on the HuggingFaceFW/fineweb sample-10BT dataset. It achieves the following results on the evaluation set:

• —Loss: 5.5467
• —Accuracy: 0.0322

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 5e-05
• —trainbatchsize: 8
• —evalbatchsize: 8
• —seed: 42
• —distributed_type: multi-GPU
• —num_devices: 2
• —totaltrainbatch_size: 16
• —totalevalbatch_size: 16
• —optimizer: Use adamwtorch with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
• —lrschedulertype: linear
• —training_steps: 200000
• —mixedprecisiontraining: Native AMP

Training results

Framework versions

• —Transformers 4.49.0.dev0
• —Pytorch 2.5.1+cu121
• —Datasets 3.2.0
• —Tokenizers 0.21.0