AdhamAshraf/image-caption-generator

Image Caption Generator (ResNet50 + LSTM)

A ResNet50 (frozen, transfer learning) + LSTM decoder model that generates natural-language captions for images. Trained on Flickr8k.

• —Full project code, training pipeline, and documentation: https://github.com/adhamashraf7788/Image-Caption-Generator
• —Live interactive demo (Hugging Face Space): https://huggingface.co/spaces/AdhamAshraf/imagecaptiongenerator

Files in this repo

vocab.json                             # vocabulary (shared across both checkpoints)
base_resnet_lstm/
├── best_model.pt                      # baseline checkpoint
└── config.yaml                        # baseline training config
resnet_lstm_regularized/
├── best_model.pt                      # regularized checkpoint (recommended -- best results)
└── config.yaml                        # regularized training config

Two checkpoints are provided:

Both checkpoints share the same vocab.json (identical vocabulary, 2,662 tokens).

Architecture

Image → ResNet50 (frozen, ImageNet-pretrained) → 2048-d feature
      → Linear(2048 → 256) projection
      → fed as first input step to a 1-layer LSTM (hidden_dim=512)
      → LSTM generates caption word-by-word (beam search recommended, width 3)

Full architecture, preprocessing, and training details: see the GitHub README.

How to use

Requires the inference code from the GitHub repo (src/inference/predict.py and its dependencies) — these checkpoints are not standalone transformers-compatible weights, they're plain PyTorch state_dicts wrapped with config metadata.

from huggingface_hub import hf_hub_download
from src.inference.predict import Predictor  # from the GitHub repo's src/

checkpoint_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="resnet_lstm_regularized/best_model.pt",
)
vocab_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="vocab.json",
)

predictor = Predictor(checkpoint_path=checkpoint_path, vocab_path=vocab_path, device="cpu")
caption = predictor.predict("path/to/image.jpg")
print(caption)

Training data

Flickr8k — 8,091 images, 5 human-written reference captions each. Split 80/10/10 (by image, not caption, to avoid leakage) using a fixed seed.

Evaluation results (test set, 810 images)

Full evaluation methodology, qualitative examples, and failure-case analysis: see the GitHub README.

Limitations

• —Trained on a small (8k image) dataset; struggles with image content/styles underrepresented in Flickr8k (predominantly people, dogs, and outdoor scenes).
• —Even the regularized model still shows some overfitting past its best epoch.
• —Generated captions are sometimes fluent but not fully grounded in image-specific detail.

See the GitHub README's Limitations section for a full discussion, including a documented failure case and how regularization + beam search improved it.