Prbhvv/mllm-recommender-api

1# Modality Evaluation Guide2 3This guide explains how to evaluate your GNN recommendation model with different modality inputs (Text-only, Image-only, and Text+Image fusion).4 5## Evaluation Metrics6 7The evaluation compares three modality configurations:8 91. **Text-only**: Uses only textual descriptions and metadata102. **Image-only**: Uses only CLIP image embeddings113. **Text+Image (Fusion)**: Uses gated fusion of both modalities12 13### Metrics Computed14 15For each modality, the following metrics are computed:16 17- **Recall@5, Recall@10**: Proportion of relevant items in top-K recommendations18- **NDCG@5, NDCG@10**: Normalized Discounted Cumulative Gain at K19- **MRR@5, MRR@10**: Mean Reciprocal Rank at K20 21## Quick Start22 23### Option 1: Evaluate Existing Model24 25If you already have a trained model (`models/best_model.pt`), you can directly evaluate it with different modality configurations:26 27```bash28python3 run_modality_evaluation.py29```30 31This will:321. Load your trained model332. Evaluate it with Text-only configuration343. Evaluate it with Image-only configuration354. Evaluate it with Fusion configuration (default)365. Print a comparison table376. Save results to `models/modality_evaluation_results.json`38 39### Option 2: Train Separate Models for Each Modality40 41For more accurate comparison, you can train separate models for each modality:42 43```bash44# Train all three models45python3 -m gnn_fusion.train_modalities --modality all46 47# Or train individual models48python3 -m gnn_fusion.train_modalities --modality text49python3 -m gnn_fusion.train_modalities --modality image50python3 -m gnn_fusion.train_modalities --modality fusion51```52 53Models will be saved in:54- `models/text/best_model.pt`55- `models/image/best_model.pt`56- `models/fusion/best_model.pt`57 58Then evaluate:59 60```bash61python3 run_modality_evaluation.py62```63 64## Understanding the Results65 66### Example Output67 68```69================================================================================70PERFORMANCE COMPARISON WITH DIFFERENT MODALITY INPUTS71================================================================================72Metrics        Recall@5  Recall@10    NDCG@5   NDCG@10     MRR@5    MRR@1073--------------------------------------------------------------------------------74Text-only        0.0584     0.0648    0.03090    0.0457    0.0238    0.029975Image-only       0.0525     0.0679    0.06689    0.0844    0.1012    0.149076Text+image       0.0620     0.0984    0.07330    0.0893    0.1100    0.159377--------------------------------------------------------------------------------78 79IMPROVEMENTS (Fusion over second-best):80  recall@5: 6.16%81  recall@10: 4.78%82  ndcg@5: 6.39%83  ndcg@10: 5.81%84  mrr@5: 8.91%85  mrr@10: 6.91%86```87 88### Interpretation89 90- **Bold values**: Best performance across all modalities91- **Improvements**: Percentage improvement of fusion over the second-best modality92- Higher values are better for all metrics93 94## Implementation Details95 96### How Modality Selection Works97 981. **FusionLayer** (in `gnn_fusion/model/gnn.py`):99   - Modified to accept a `modality` parameter100   - `modality='text'`: Uses only text projection101   - `modality='image'`: Uses only image projection102   - `modality='fusion'`: Uses gated fusion (default)103 1042. **Data Loading** (in `gnn_fusion/data_loader.py`):105   - Stores raw image and text embeddings separately106   - `data['item'].x_image`: Raw CLIP image embeddings (512-D)107   - `data['item'].x_text`: Raw text embeddings (768-D)108   - `data['item'].x`: Fused features (256-D)109 1103. **Evaluation** (in `gnn_fusion/evaluate_modalities.py`):111   - Loads the trained model112   - For each modality:113     - Applies appropriate feature masking114     - Runs forward pass through GNN115     - Computes metrics on test set116   - Aggregates and compares results117 118### Metric Calculations119 120**Recall@K**:121```python122recall = (# of relevant items in top-K) / (total # of relevant items)123```124 125**NDCG@K**:126```python127DCG = sum(rel_i / log2(i + 1)) for i in top-K128IDCG = ideal DCG (all relevant items at top)129NDCG = DCG / IDCG130```131 132**MRR@K**:133```python134MRR = 1 / (rank of first relevant item in top-K)135```136 137## Customization138 139### Modify K Values140 141Edit `gnn_fusion/evaluate_modalities.py`:142 143```python144results = evaluator.evaluate_all_modalities(test_loader, k_list=[5, 10, 20])145```146 147### Change Test Set148 149The evaluation uses the test split defined in `data/splits/test.txt`. To use a different split:150 151```python152from gnn_fusion.data_loader import SessionGraphDataset153 154dataset = SessionGraphDataset(config.data_dir, split='val')  # or 'train'155```156 157### Add New Modalities158 159To add a new modality combination (e.g., "text + metadata"):160 1611. Modify `FusionLayer.forward()` to handle the new modality1622. Update `evaluate_modality()` to apply appropriate feature selection1633. Add the modality to the evaluation loop164 165## Troubleshooting166 167### Issue: "No positive items found"168 169**Solution**: Ensure your test set has session-item interactions:170```bash171head -n 20 data/splits/test.txt172```173 174### Issue: "CUDA out of memory"175 176**Solution**: Reduce batch size or use CPU:177```python178# In config.py179device: str = "cpu"180```181 182### Issue: "Model not found"183 184**Solution**: Train a model first:185```bash186python3 -m gnn_fusion.train187```188 189## Expected Performance190 191Based on similar multimodal recommendation systems:192 193- **Text-only**: Good for content-based recommendations, but misses visual patterns194- **Image-only**: Captures visual similarity well, especially for games with distinctive art styles195- **Fusion**: Best overall performance by combining complementary information196 197Typical improvements of fusion over single modalities: **5-10%** across metrics.198 199## Citation200 201If you use this evaluation framework, please cite:202 203```bibtex204@article{your_paper,205  title={Multimodal Graph Neural Networks for Session-based Recommendation},206  author={Your Name},207  journal={Your Conference/Journal},208  year={2025}209}210```211 212## Support213 214For issues or questions:2151. Check the logs in `models/modality_evaluation_results.json`2162. Review the model architecture in `gnn_fusion/model/gnn.py`2173. Verify data preprocessing in `gnn_fusion/data_loader.py`218