sumitdotml/seq2seq-de-en

1---2license: mit3datasets:4- wmt/wmt195language:6- en7- de8pipeline_tag: translation9---10 11# Seq2Seq German-English Translation Model12 13A sequence-to-sequence neural machine translation model that translates German text to English, built using PyTorch with LSTM encoder-decoder architecture.14 15## Model Description16 17This model implements the classic seq2seq architecture from [Sutskever et al. (2014)](https://arxiv.org/abs/1409.3215) for German-English translation:18 19- **Encoder**: 2-layer LSTM that processes German input sequences20- **Decoder**: 2-layer LSTM that generates English output sequences  21- **Training Strategy**: Teacher forcing during training, autoregressive generation during inference22- **Vocabulary**: 30k German words, 25k English words23- **Dataset**: Trained on 2M sentence pairs from WMT19 (subset of full 35M dataset)24 25## Model Architecture26 27```28German Input → Embedding → LSTM Encoder → Context Vector → LSTM Decoder → Embedding → English Output29```30 31**Hyperparameters:**32- Embedding size: 25633- Hidden size: 51234- LSTM layers: 2 (both encoder/decoder)35- Dropout: 0.336- Batch size: 6437- Learning rate: 0.000338 39## Training Data40 41- **Dataset**: WMT19 German-English Translation Task42- **Size**: 2M sentence pairs (filtered subset)43- **Preprocessing**: Sentences filtered by length (5-50 tokens)44- **Tokenization**: Custom word-level tokenizer with special tokens (`<PAD>`, `<UNK>`, `<START>`, `<END>`)45 46## Performance47 48**Training Results (5 epochs):**49- Initial Training Loss: 4.0949 → Final: 3.1843 (91% improvement)50- Initial Validation Loss: 4.1918 → Final: 3.8537 (34% improvement)51- Training Device: Apple Silicon (MPS)52 53## Usage54 55### Quick Start56 57```python58# This is a custom PyTorch model, not a Transformers model59# Download the files and use with the provided inference script60 61import requests62from pathlib import Path63 64# Download model files65base_url = "https://huggingface.co/sumitdotml/seq2seq-de-en/resolve/main"66files = ["best_model.pt", "german_tokenizer.pkl", "english_tokenizer.pkl"]67 68for file in files:69    response = requests.get(f"{base_url}/{file}")70    Path(file).write_bytes(response.content)71    print(f"Downloaded {file}")72```73 74### Translation Examples75 76```bash77# Interactive mode78python inference.py --interactive79 80# Single translation81python inference.py --sentence "Hallo, wie geht es dir?" --verbose82 83# Demo mode84python inference.py85```86 87**Example Translations:**88- `"Das ist ein gutes Buch."` → `"this is a good idea."`89- `"Wo ist der Bahnhof?"` → `"where is the <UNK>"`90- `"Ich liebe Deutschland."` → `"i share."`91 92## Files Included93 94- `best_model.pt`: PyTorch model checkpoint (trained weights + architecture)95- `german_tokenizer.pkl`: German vocabulary and tokenization logic96- `english_tokenizer.pkl`: English vocabulary and tokenization logic97 98## Installation & Setup99 1001. **Clone the repository:**101   ```bash102   git clone https://github.com/sumitdotml/seq2seq103   cd seq2seq104   ```105 1062. **Set up environment:**107   ```bash108   uv venv && source .venv/bin/activate  # or python -m venv .venv109   uv pip install torch requests tqdm    # or pip install torch requests tqdm110   ```111 1123. **Download model:**113   ```bash114   python scripts/download_pretrained.py115   ```116 1174. **Start translating:**118   ```bash119   python scripts/inference.py --interactive120   ```121 122## Model Architecture Details123 124The model uses a custom implementation with these components:125 126- **Encoder** (`src/models/encoder.py`): LSTM-based encoder with embedding layer127- **Decoder** (`src/models/decoder.py`): LSTM-based decoder with attention-free architecture  128- **Seq2Seq** (`src/models/seq2seq.py`): Main model combining encoder-decoder with generation logic129 130## Limitations131 132- **Vocabulary constraints**: Limited to 30k German / 25k English words133- **Training data**: Only 2M sentence pairs (vs 35M in full WMT19)134- **No attention mechanism**: Basic encoder-decoder without attention135- **Simple tokenization**: Word-level tokenization without subword units136- **Translation quality**: Suitable for basic phrases, struggles with complex sentences137 138## Training Details139 140**Environment:**141- Framework: PyTorch 2.0+142- Device: Apple Silicon (MPS acceleration)  143- Training time: ~5 epochs144- Validation strategy: Hold-out validation set145 146**Optimization:**147- Optimizer: Adam (lr=0.0003)148- Loss function: CrossEntropyLoss (ignoring padding)149- Gradient clipping: 1.0150- Scheduler: StepLR (step_size=3, gamma=0.5)151 152## Reproduce Training153 154```bash155# Full training pipeline156python scripts/data_preparation.py      # Download WMT19 data157python src/data/tokenization.py        # Build vocabularies  158python scripts/train.py                # Train model159 160# For full dataset training, modify data_preparation.py:161# use_full_dataset = True  # Line 133-134162```163 164## Citation165 166If you use this model, please cite:167 168```bibtex169@misc{seq2seq-de-en,170  author = {sumitdotml},171  title = {German-English Seq2Seq Translation Model},172  year = {2025},173  url = {https://huggingface.co/sumitdotml/seq2seq-de-en},174  note = {PyTorch implementation of sequence-to-sequence translation}175}176```177 178## References179 180- Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. NeurIPS.181- WMT19 Translation Task: https://huggingface.co/datasets/wmt/wmt19182 183## License184 185MIT License - See repository for full license text.186 187## Contact188 189For questions about this model or training code, please open an issue in the [GitHub repository](https://github.com/sumitdotml/seq2seq).