Apppublic

RaviNaik/MultiModal-Phi2

sourceHugging Facemitupdated 3y agoView on Hugging Face

4likes

App README

Phi2 : Multimodal Finetuning

Details

1.LLM Backbone: Phi2
2.Vision Tower: clip-vit-large-patch14-336
3.Audio Model: Whisper
4.Pretraining Dataset: LAION-CC-SBU dataset with BLIP captions(200k samples)
5.Finetuning Dataset: Instruct 150k dataset based on COCO

Design

Pretraining

Training Loss Curve

Learing Rate

Training Logs

Finetuning

Training Loss Curve

Learing Rate

Training Logs

Results