pratik-250620/MultiModal-Coherence-AI

Multimodal Coherence AI

Generate semantically coherent text + image + audio bundles and evaluate cross-modal alignment using the Multimodal Semantic Coherence Index (MSCI).

How it works

1. 1.Text — generated via HF Inference API
2. 2.Image — retrieved from a curated index using CLIP (ViT-B/32) embeddings
3. 3.Audio — retrieved from a curated index using CLAP (HTSAT-unfused) embeddings
4. 4.MSCI — computed as 0.45 * cos_sim(text, image) + 0.45 * cos_sim(text, audio)

Research

This demo accompanies a study evaluating multimodal semantic coherence across three research questions:

• —RQ1: Is MSCI sensitive to controlled semantic perturbations? (Supported, d > 2.0)
• —RQ2: Does structured planning improve cross-modal alignment? (Not supported)
• —RQ3: Does MSCI correlate with human coherence judgments? (Supported, rho = 0.379)