antflydb/clipclap

CLIPCLAP — Unified Text + Image + Audio Embeddings

CLIPCLAP is a unified multimodal embedding model that maps text, images, and audio into a shared 512-dimensional vector space. It combines OpenAI's CLIP (text + image) with LAION's CLAP (audio) through a trained linear projection.

Built by antflydb for use with Antfly Inference, a standalone ML inference service for embeddings, chunking, reranking, and local model serving.

Architecture

Text  ──→ CLIP text encoder  ──→ text_projection  ──→ 512-dim (CLIP space)
Image ──→ CLIP visual encoder ──→ visual_projection ──→ 512-dim (CLIP space)
Audio ──→ CLAP audio encoder  ──→ audio_projection  ──→ 512-dim (CLIP space)

• —Text & Image: Standard CLIP ViT-B/32 encoders and projections (unchanged from openai/clip-vit-base-patch32).
• —Audio: CLAP HTSAT audio encoder from laion/larger_clap_music_and_speech. The audio projection combines CLAP's native audio projection (1024→512) with a trained 512→512 linear layer that maps CLAP audio space into CLIP space.

All three modalities produce 512-dimensional L2-normalized embeddings that are directly comparable via cosine similarity.

Intended Uses

• —Multimodal search (text↔image↔audio)
• —Building unified media indexes with Antfly
• —Cross-modal retrieval (find images from audio queries, audio from text, etc.)
• —Audio-visual content discovery

How to Use with Antfly Inference

# Pull and run the model
antfly inference pull antflydb/clipclap:gguf:Q4_K
antfly inference run

# Embed text
curl -X POST http://localhost:8082/embed \
  -H "Content-Type: application/json" \
  -d '{
    "model": "clipclap",
    "input": [
      {"type": "text", "text": "a cat sitting on a windowsill"},
      {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
      {"type": "audio_url", "audio_url": {"url": "https://example.com/cat-purring.wav"}}
    ]
  }'

Training Details

Audio Projection

The audio projection layer bridges CLAP and CLIP embedding spaces. Training procedure:

1. 1.Load audio-caption pairs from OpenSound/AudioCaps
2. 2.Encode audio through CLAP: audio encoder → audio_projection → L2 normalize
3. 3.Encode captions through CLIP: text encoder → text_projection → L2 normalize
4. 4.Train a 512→512 linear projection (CLAP audio → CLIP text) using CLIP-style contrastive loss (InfoNCE)

The contrastive loss pushes matching audio-text pairs together while pushing non-matching pairs apart within each batch, preserving content discrimination.

Hyperparameters

Source Models

ONNX Files

Additional files: clip_config.json, tokenizer.json, preprocessor_config.json, projection_training_metadata.json.

Limitations

• —Audio duration: Audio is truncated to ~10 seconds (inherited from CLAP)
• —Language: Primarily English text support
• —Audio-visual alignment: The projection is trained via caption similarity (audio↔text↔image), not direct audio-image pairs. Audio-to-image retrieval may be less precise than text-to-image.
• —CLIP limitations: Inherits CLIP's weaknesses in fine-grained visual classification, object counting, and abstract concepts
• —Training data: Audio projection trained on AudioCaps which covers common environmental sounds and may underperform on niche audio domains

Citation

If you use CLIPCLAP, please cite the underlying models:

@inproceedings{radford2021clip,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and others},
  booktitle={ICML},
  year={2021}
}

@inproceedings{wu2023clap,
  title={Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
  author={Wu, Yusong and Chen, Ke and Zhang, Tianyu and others},
  booktitle={ICASSP},
  year={2023}
}