hrezaei/nanoGPTLookAhead

This model is part of the work published in the paper Interactive Text Games: Lookahead Is All You Need!

Four models are introduced in the above paper:

• —nanoGPTLA
• —nanoGPTLAA
• —nanoGPTLAA2
• —nanoGPTLAE

These models are implemented in this repository which is a customized version of nanoGPT.

The same variations are also implemented in this fork of Transformers library, on top of Google-t5/T5 implementation. These models are also trained and published as follows:

• —T5LA
• —T5LAA
• —T5LAA2
• —T5LAE

All the above models are on the scale of GPT2 (~100M parameters). The work is in progress to train them on larger scales.

This model has been pushed to the Hub using the PytorchModelHubMixin integration:

• —Library: https://github.com/HRezaei/nanoGPT
• —Docs: [More Information Needed]