pratik-250620/MultiModal-Coherence-AI

1---2title: Multimodal Coherence AI3emoji: "\U0001f3a8"4colorFrom: purple5colorTo: pink6sdk: streamlit7sdk_version: "1.41.0"8app_file: app.py9pinned: false10license: mit11short_description: Coherent text + image + audio with MSCI12---13 14# Multimodal Coherence AI15 16Generate semantically coherent **text + image + audio** bundles and evaluate17cross-modal alignment using the **Multimodal Semantic Coherence Index (MSCI)**.18 19## How it works20 211. **Text** — generated via HF Inference API222. **Image** — retrieved from a curated index using CLIP (ViT-B/32) embeddings233. **Audio** — retrieved from a curated index using CLAP (HTSAT-unfused) embeddings244. **MSCI** — computed as `0.45 * cos_sim(text, image) + 0.45 * cos_sim(text, audio)`25 26## Research27 28This demo accompanies a study evaluating multimodal semantic coherence across29three research questions:30 31- **RQ1**: Is MSCI sensitive to controlled semantic perturbations? (Supported, d > 2.0)32- **RQ2**: Does structured planning improve cross-modal alignment? (Not supported)33- **RQ3**: Does MSCI correlate with human coherence judgments? (Supported, rho = 0.379)34