alexgara/lstm-en-es-translator

LSTM English-to-Spanish Translator

A sequence-to-sequence neural machine translation model built entirely from scratch — custom LSTM cells, encoder, decoder, attention mechanism, and beam search — as a deep learning educational project.

Code: github.com/alexgarabt/lstm-translator

<video src="https://huggingface.co/alexgara/lstm-en-es-translator/resolve/main/img/lstm.mp4" controls width="600"></video>

Architecture

Source (EN) → [Embedding] → [BiLSTM Encoder] → encoder_outputs
                                                      ↓
                                              [Dot-Product Attention]
                                                      ↓
Target (ES) → [Embedding] → [LSTM Decoder] → [Output Projection] → predicted tokens

Quick Start

pip install torch sentencepiece huggingface_hub

import torch
from huggingface_hub import hf_hub_download

REPO_ID = "alexgara/lstm-en-es-translator"

# Download
checkpoint_path = hf_hub_download(REPO_ID, "model.pt")
en_tok_path = hf_hub_download(REPO_ID, "spm_en.model")
es_tok_path = hf_hub_download(REPO_ID, "spm_es.model")

# For full usage with the translator package, see the GitHub repo

For full inference with greedy/beam decoding, clone the GitHub repo and run:

uv run python scripts/inference.py --interactive

Example Translations

Beam search eliminates the repetition artifacts visible in greedy decoding.

Performance

The model (~31.9M parameters) runs on both CPU and GPU. No GPU required.

Model weights in float32: 31.9M params x 4 bytes = ~128 MB, plus tokenizers and PyTorch overhead.

# Force CPU inference
uv run python scripts/inference.py --device cpu --interactive

Training

Hyperparameters

Training Curves

<table> <tr> <td><img src="img/epochtrainloss.svg" width="400" alt="Train Loss"/></td> <td><img src="img/epochvalloss.svg" width="400" alt="Val Loss"/></td> </tr> <tr> <td align="center">Train Loss (epoch)</td> <td align="center">Validation Loss (epoch)</td> </tr> </table>

<table> <tr> <td><img src="img/trainloss.svg" width="270" alt="Train Loss (step)"/></td> <td><img src="img/traingradnorm.svg" width="270" alt="Gradient Norm"/></td> <td><img src="img/trainattention_entropy.svg" width="270" alt="Attention Entropy"/></td> </tr> <tr> <td align="center">Train Loss (step)</td> <td align="center">Gradient Norm</td> <td align="center">Attention Entropy</td> </tr> </table>

The apparent uptick in train loss after epoch ~16 is caused by teacher forcing decay (the training task gets harder as the model relies more on its own predictions). The validation loss — always evaluated fully autoregressively — decreases monotonically.

Attention Visualization

The model learns interpretable word alignments:

<table> <tr> <td><img src="img/context1.png" width="270"/></td> <td><img src="img/context2.png" width="270"/></td> <td><img src="img/context3.png" width="270"/></td> </tr> </table>

Dataset

What's Built From Scratch

Every neural network component is implemented from first principles — no torch.nn.LSTM or pre-built seq2seq modules:

• —LSTMCell — fused gates, Xavier init, forget bias = 1.0
• —BiLSTM Encoder — forward + backward with learned projection
• —Dot-Product Attention — score, mask, softmax, context
• —LSTM Decoder — step-by-step with attention and teacher forcing
• —Beam Search — k-best decoding with length normalization

Files in This Repo

Limitations

• —Best for short-to-medium sentences (under 30 words)
• —English → Spanish only
• —LSTM architecture is inherently sequential (slower inference than Transformers)
• —Trained on conversational + parliamentary text — may struggle with specialized domains

License

MIT

Citation

@misc{lstm-en-es-translator,
  author = {Alex Gara},
  title = {LSTM English-to-Spanish Translator},
  year = {2025},
  url = {https://github.com/alexgarabt/lstm-translator}
}