Modelpublic

bilalfaye/OneEncoder-text-image-audio

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes8downloads

Model Card

🖼️📝 OneEncoder: A Unified Text & Image & Audio Model

OneEncoder is a lightweight framework for cross-modal alignment, focusing on efficiently integrating text, image and audio (with future extensions to other modalities). Unlike traditional methods relying on massive modality-specific encoders, OneEncoder progressively aligns different data types, making it cost-effective and performant even on small paired datasets.

🚀 Key Features

✅ Multimodal Alignment: Initially supports text & image & audio, with extension to other modalities. ✅ Lightweight & Efficient: Avoids full retraining when adding new modalities. ✅ Superior Performance: Outperforms models that require large specialized datasets.

🎯 Applications

—Visual Question Answering (VQA)
—Image-Text-Audio Retrieval
—Multimodal Content Understanding

📄 Research Paper

📜 arXiv: OneEncoder: Progressive Cross-Modal Alignment

📌 Resources

🔗 GitHub Repo: OneEncoder 🚀 Hugging Face Demo: OneEncoder Retriever 📓 Demo Notebook: OneEncoder Demos 🔊 OneEncoder for Text, Image: HF Model 🔊 OneEncoder for Text, Image & Video: HF Model 🔊 OneEncoder for Text, Image & X-ray: HF Model

📝 Authors

📌 Bilal FAYE, Hanane AZZAG, Mustapha LEBBAH, Djamel BOUCHAFFRA

Note: This model is training with temperature=2.5 and addition as fusion operation