KeerthiSaiKasaraneni/NeuralNetworkLanguageModel

LSTM-Based Language Model for Next Token Prediction 🧠

This repository hosts an LSTM-based neural network model designed for next token prediction in both English and Danish languages. The model employs a non-transformer-based LSTM architecture, enabling it to capture temporal dependencies within text data effectively.

🌟 Key Features

• —Multilingual Support: Capable of generating text in English and Danish.
• —LSTM Architecture: Utilizes Long Short-Term Memory layers to learn from the sequence of words.
• —Custom Tokenization: Features a word-level tokenizer to process text into tokens.
• —Checkpointing: Includes functionality to save training checkpoints, aiding in model evaluation and training resumption.
• —Perplexity Evaluation: Employs perplexity scores to measure prediction capabilities, aiming for lower scores to indicate better performance.

🚀 Quick Start

This model is equipped with a Gradio web interface, allowing users to interactively generate text sequences by providing a starting phrase in either language.

Text Generation Examples

• —English Input: "Once upon a time"
• —Generated Text: "Once upon a time as the argument is prime straight panel of water and plot..."

• —Danish Input: "En gang"
• —Generated Text: "En gang variable og cytokines stærke follow netflix skove elefant formåede..."

📊 Model Performance

The final validation perplexity of the model stands at 2409.41, reflecting its ability to predict the next token in a sequence.

🏗 Model Architecture

1. 1.Embedding Layer: Converts words into dense vectors of a fixed size.
2. 2.LSTM Layers: Two layers that capture the word dependencies in sequences.
3. 3.Fully Connected Layer: Outputs probabilities over the vocabulary for next word prediction.

Parameters

• —Embedding Dimension: 128
• —Hidden Units: 256
• —LSTM Layers: 2
• —Dropout Rate: 0.3
• —Learning Rate: 0.002
• —Batch Size: 32
• —Context Window: 10

🔧 Training

Training involved 2000 iterations with checkpoints at every 100 steps. The model uses datasets in English and Danish, structured to facilitate efficient learning.

📁 Repository Contents

• —finallstmmodel.pth: The trained model in PyTorch format.
• —English.json: Training dataset in English.
• —Danish.json: Training dataset in Danish.
• —app.py: Gradio application script for model interaction.
• —README.md: Detailed project documentation.

License

This project is available under the MIT license.

Feel free to clone this repository to train the model further or to develop it according to your specific needs!