arykarki/seq2seq-en-es

Neural Machine Translation with Attention 🚀

A PyTorch implementation of a Sequence-to-Sequence model with Attention for English-Spanish translation.

🌟 Features

• —Bidirectional GRU Encoder: Captures context from both directions of the input sequence
• —Attention Mechanism: Helps the model focus on relevant parts of the input sequence
• —Teacher Forcing: Implements curriculum learning for better training stability
• —Dynamic Batching: Efficient training with variable sequence lengths
• —Hugging Face Integration: Uses MarianTokenizer for robust text processing

🏗️ Architecture

The model consists of three main components:

1. 1.Encoder: Bidirectional GRU network that processes input sequences
2. 2.Attention: Computes attention weights for each encoder state
3. 3.Decoder: GRU network that generates translations using attention context

Input → Encoder → Attention → Decoder → Translation
      ↑          ↑          ↑
      Embeddings Context    Attention Weights

🚀 Quick Start

1. 1.Clone the repository:

git clone https://github.com/yourusername/nmt-attention.git
cd nmt-attention

1. 1.Install dependencies:

pip install torch transformers datasets

1. 1.Train the model:

python train.py

1. 1.Translate text:

from translate import translate
text = "How are you?"
translated = translate(model, text, tokenizer)
print(translated)

# Loading a saved model
model = Seq2Seq(encoder, decoder, device)
model.load_state_dict(torch.load('LSTM_text_generator.pth'))
model.eval()

📊 Model Performance

Training metrics after 10 epochs:

• —Initial Loss: 11.147
• —Final Loss: 3.527
• —Training Time: ~2 hours on NVIDIA V100

🔧 Hyperparameters

BATCH_SIZE = 32
LEARNING_RATE = 1e-3
CLIP = 1.0
N_EPOCHS = 10
ENC_EMB_DIM = 256
DEC_EMB_DIM = 256
ENC_HID_DIM = 512
DEC_HID_DIM = 512

📚 Dataset

Using the loresiensis/corpus-en-es dataset from Hugging Face Hub, which provides English-Spanish sentence pairs for training.

🤝 Contributing

1. 1.Fork the repository
2. 2.Create your feature branch (git checkout -b feature/amazing-feature)
3. 3.Commit your changes (git commit -m 'Add amazing feature')
4. 4.Push to the branch (git push origin feature/amazing-feature)
5. 5.Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• —Attention Is All You Need paper
• —Hugging Face for the transformers library and datasets
• —PyTorch team for the amazing deep learning framework

⭐️ If you found this project helpful, please consider giving it a star!