Modelpublic

gvij/minicpm-av-music-avqa

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes10downloads

Model Card

MiniCPM-AV Music-AVQA Model

This is a fine-tuned MiniCPM-AV model trained on the MUSIC-AVQA-v2.0 dataset for audio-visual question answering.

Model Details

—Base Model: openbmb/MiniCPM-V
—Audio Encoder: UsefulSensors/moonshine-tiny
—Training Dataset: DraculaDragon/MUSIC-AVQA-v2.0
—Training Config:
—Epochs: 3
—Batch Size: 64
—Learning Rate: 2e-5
—LoRA Rank: 8
—LoRA Alpha: 16
—Mixed Precision: bf16
—Best Validation Loss: 0.1525

Files

—audio_components.pt: Audio projector, compressor, and modality embeddings
—adapter_config.json: LoRA adapter configuration
—adapter_model.safetensors: LoRA adapter weights (13.6M parameters)

Usage

python

from src.modeling_minicpm_av import MiniCPMAV, MiniCPMAVConfig
import torch

# Load model
config = MiniCPMAVConfig()
model = MiniCPMAV(config=config)

# Load pretrained weights
model.load_pretrained("gvij/minicpm-av-music-avqa")
model.eval()

# Prepare inputs
audio = torch.randn(16000)  # 1 second of audio
image = PIL.Image.open('image.jpg')
question = "What instrument is playing?"

# Generate answer
answer = model.generate_with_audio(
    images=image,
    audio=audio,
    question=question
)
print(answer)

Citation

bibtex

@misc{minicpm-av,
  title={MiniCPM-AV: Audio-Visual Question Answering with MiniCPM-V},
  author={Gaurav Vij},
  year={2025}
}