idosumit/seq2seq

Classic Seq2Seq (Sutskever et al. 2014) Reimplementation

This is a faithful implementation of the original sequence-to-sequence paper by Sutskever et al. (2014), "Sequence to Sequence Learning with Neural Networks", trained for German-to-English translation on the WMT19 dataset.

Model Description

• —Architecture: 4-layer deep LSTM encoder-decoder with 1000 hidden units per layer
• —Parameters: 194 million parameters
• —Training Data: WMT19 German-English dataset (35 million sentence pairs)
• —Training Duration: 10 epochs
• —Key Innovation: Input sequence reversal (crucial Sutskever et al. finding)
• —Tokenization: SentencePiece (50k German vocab, 40k English vocab)
• —Decoding: Beam search with length normalization

Model Details

Architecture

This model implements the exact architecture from the original Sutskever et al. paper:

• —Encoder: 4-layer LSTM with input sequence reversal
• —Decoder: 4-layer LSTM with teacher forcing during training
• —No attention mechanism - pure encoder-decoder as originally conceived
• —Beam search decoding with configurable beam size (default: 12)

Training Hyperparameters I Used

• —Optimizer: SGD with momentum (lr=0.7, momentum=0.9)
• —Loss: Cross-entropy with padding token masking
• —Regularization: Dropout (0.2) and gradient clipping (5.0)
• —Hardware Used: Trained on NVIDIA A6000 (10 vCPUs, 60GB RAM, 200GB disk)
• —Training Time: Roughly 60 hours on 500k dataset subset

Usage

Installation

pip install torch sentencepiece requests tqdm

Quick Start

# Download the model files
python -c "
import requests
from pathlib import Path

files = [
    'best_model.pt',
    'german_sp.model', 
    'english_sp.model',
    'german_sp.vocab',
    'english_sp.vocab'
]

base_url = 'https://huggingface.co/idosumit/seq2seq/resolve/main'
for file in files:
    response = requests.get(f'{base_url}/{file}')
    Path(file).write_bytes(response.content)
    print(f'Downloaded {file}')
"

Using with the Full Codebase

For the complete training and inference pipeline, clone the full repository:

git clone https://gitlab.com/sumitdoesml/classic-seq2seq
cd classic-seq2seq
uv venv && source .venv/bin/activate
pip install -r requirements.txt

# Download pre-trained model
python scripts/download_pretrained.py

# Interactive translation
python scripts/inference.py --interactive --device cpu

# Single sentence translation
python scripts/inference.py --sentence "Hallo, wie geht es dir?" --device cpu

Example Translations

German: "Hallo, wie geht es dir?"
English: "Hello, how are you doing?"

German: "Können Sie mir bitte helfen?"
English: "Can you help me, please?"

German: "Das Wetter ist heute schön."
English: "The weather is beautiful today."

Technical Specifications

• —Framework: PyTorch
• —Precision: FP32 (with optional mixed precision training)
• —Memory Requirements: ~6GB VRAM for inference, 16GB+ for training
• —Input Length: Up to 100 tokens (filtered during training)
• —Languages: German → English only

Limitations

• —Era: This is a 2014-era model without modern improvements like attention
• —Repetition: May exhibit repetition patterns typical of early seq2seq models
• —Vocabulary: Limited to SentencePiece vocabularies (50k German, 40k English)
• —Domain: Trained on general domain text, may not perform well on specialized domains
• —Unidirectional: German to English only, not bidirectional

Training Details

Hyperparameters

• —Learning Rate: 0.7 with StepLR decay (gamma=0.5, step_size=3)
• —Batch Size: 16 (memory-constrained)
• —Gradient Clipping: 5.0
• —Dropout: 0.2
• —Sequence Length: 3-100 tokens

Dataset

• —Source: WMT19 German-English
• —Size: 35 million sentence pairs
• —Preprocessing: Length filtering, quality checks, automatic train/val split
• —Tokenization: SentencePiece trained on the full dataset

Repository

Full source code, training scripts, and documentation: GitLab Repository

Note: This is an educational/research implementation to understand foundational seq2seq concepts. For production German-English translation, consider modern transformer-based models with attention mechanisms.