BesoNe-69/rec-foundation-model

Netflix Foundation Model for Personalized Recommendation - POC

A proof-of-concept implementation of Netflix's Foundation Model approach for recommendations using the MovieLens 20M dataset.

Architecture Highlights

This implementation follows Netflix's blog post on Foundation Models for Personalized Recommendation:

1. 1.Heterogeneous Embeddings: Combines learnable item ID embeddings with metadata (genre) embeddings using a learned fusion gate
2. 2.Causal Transformer: Decoder-only transformer architecture for next-token prediction
3. 3.Cold-Start Handling: Metadata-based embeddings allow recommendations for new items

Project Structure

rec-foundation-model/
├── src/                    # Core implementation
│   ├── data_processing.py  # Tokenization & dataloaders
│   ├── model.py           # Foundation model architecture
│   ├── train.py           # Training loop
│   └── evaluate.py        # Evaluation metrics
├── config/
│   └── config.yaml        # All hyperparameters
├── notebooks/
│   └── train_on_colab.ipynb  # GPU training notebook
├── checkpoints/           # Model checkpoints
├── logs/                  # Training logs
├── data/                  # MovieLens 20M dataset
├── main.py               # Entry point
├── requirements.txt      # Dependencies
└── README.md             # This file

Setup

1. Install Dependencies

pip install -r requirements.txt

2. Download MovieLens 20M Dataset

wget https://files.grouplens.org/datasets/movielens/ml-20m.zip
unzip ml-20m.zip
mkdir -p data
mv ml-20m data/

3. Configure

Edit config/config.yaml to adjust:

• —Model architecture (embedding dim, layers, heads)
• —Training hyperparameters (batch size, learning rate, epochs)
• —Data paths

Running the POC

Local Training (CPU)

python main.py

GPU Training (Google Colab)

1. 1.Upload notebooks/train_on_colab.ipynb to Colab
2. 2.Enable GPU runtime (Runtime → Change runtime type → GPU)
3. 3.Run all cells
4. 4.Results saved to Google Drive

Model Architecture

Input: User interaction sequence
  ↓
[Item ID Embedding] + [Genre Metadata Embedding]
  ↓
Fusion Gate (learns α)
  ↓
Combined Embedding = α * ID_emb + (1-α) * Meta_emb
  ↓
+ Positional Encoding
  ↓
Causal Transformer (4 layers, 8 heads)
  ↓
Prediction Head
  ↓
Output: Next item logits

Key Implementation Details

1. 1.Tokenization: Filters ratings ≥4.0 as positive interactions, creates temporal sequences
2. 2.Causal Masking: Ensures model only sees past interactions during training
3. 3.Fusion Gate: Allows model to automatically balance between ID and metadata embeddings
4. 4.Temporal Splitting: Uses last interactions for testing, maintaining temporal ordering

Evaluation Metrics

• —Hit Rate@K: Percentage of times the true item appears in top-K recommendations
• —NDCG@K: Normalized Discounted Cumulative Gain at K
• —MRR@K: Mean Reciprocal Rank at K

Evaluated at K = [5, 10, 20]

Expected Results

On MovieLens 20M, you should expect:

• —Hit Rate@10: 0.40-0.50
• —NDCG@10: 0.25-0.35
• —MRR@10: 0.20-0.30

Training time: ~2-4 hours on a single T4 GPU

Configuration

Key hyperparameters in config/config.yaml:

model:
  embed_dim: 256        # Embedding dimension
  n_heads: 8           # Number of attention heads
  n_layers: 4          # Number of transformer layers
  dropout: 0.1         # Dropout rate

training:
  batch_size: 128      # Batch size
  num_epochs: 20       # Number of epochs
  learning_rate: 0.0001  # Learning rate

Customization

Adding More Metadata

Extend _process_metadata() in src/data_processing.py to include:

• —Release year
• —Director/cast information
• —Tags

Multi-Objective Loss

Add auxiliary prediction heads in src/model.py:

self.genre_head = nn.Linear(embed_dim, num_genres)

Increasing Scale

Adjust in config/config.yaml:

model:
  embed_dim: 512
  n_layers: 8
  n_heads: 16

Technical Details

Data Processing

• —Vocabulary Size: ~27K movies (from MovieLens 20M)
• —Sequence Length: Max 50 interactions per user
• —Train/Val/Test Split: Temporal split (last item for test)

Model Specifications

• —Parameters: ~15M trainable parameters
• —Architecture: Transformer encoder with causal masking
• —Optimization: AdamW with OneCycleLR scheduler
• —Regularization: Dropout, gradient clipping, weight decay

References

• —Netflix Foundation Model Blog Post
• —MovieLens 20M Dataset
• —Attention Is All You Need

Future Work

• —[ ] Add multi-objective loss (genre prediction)
• —[ ] Implement KV caching for inference
• —[ ] Add time-based features (day-of-week, time-of-day)
• —[ ] Experiment with larger models (8 layers, 512d)
• —[ ] Deploy inference API with FastAPI
• —[ ] Add user cold-start handling
• —[ ] Implement incremental training

Author

Neha Amin

Note: This is a proof-of-concept implementation for educational purposes. For production use, additional optimizations and safety measures would be required. ---

Training Results (POC)

Completed: January 25, 2025 on Google Colab T4 GPU

Quick Results

Model: 2 layers, 128d embeddings, 5 epochs (~40 min training)

Status: ✅ Architecture validated, ready to scale to full model

See training_results.md for detailed analysis.