Modelpublic

AL-GR/Forge-EMB-mmclip

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

Model Card

Forge-EMB-mmclip

This repository contains a multimodal embedding model based on CLIP, designed for the FORGE framework tasks. It allows for joint embedding of images and text.

The model was introduced in the paper FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets.

—Code: selous123/al_sid
—Framework: AL-GR/FORGE

📚 Reference: This implementation is based on the mm_clip project from the al_sid repository.

🚀 Quick Start

The easiest way to use this model is by running the provided Jupyter Notebook.

1. Clone the Repository

bash

git clone https://huggingface.co/AL-GR/Forge-EMB-mmclip
cd Forge-EMB-mmclip

⚠️ Important Note on Large Files: The file clip.pth is stored using Git LFS. If you see that clip.pth is only ~134 Bytes on your local machine, it means the actual weights were not downloaded. Please ensure you have git-lfs installed and run: ``bash git lfs install git lfs pull `` Or download the files directly via the "Download" button on the website.

2. Install Dependencies

Ensure you have the necessary libraries installed (e.g., torch, transformers, jupyter).

bash

pip install torch torchvision transformers matplotlib pillow

3. Run the Demo

Simply open and run the notebook:

bash

jupyter notebook CLIP_demo.ipynb

Inside the notebook, you will find examples of how to load the model from clip.pth and process images like test.jpg.

📂 File Structure

File/Folder	Description
`CLIP_demo.ipynb`	The main entry point. Contains inference code and visualization examples.
`clip.pth`	The pre-trained model weights (PyTorch checkpoint).
`bert/`	Configuration or tokenizer files related to the text encoder.
`utils/`	Helper functions used by the demo.
`test.jpg`	A sample image for testing the model.

🔗 Related Resources

—Original Implementation: selous123/al_sid/mm_clip
—FORGE Framework: AL-GR/FORGE

📄 Citation

If you find this work helpful, please cite:

bibtex

@article{fu2025forge,
  title={FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets},
  author={Fu, Kairui and Zhang, Tao and Xiao, Shuwen and Wang, Ziyang and Zhang, Xinming and Zhang, Chenchi and Yan, Yuliang and Zheng, Junjun and others},
  journal={arXiv preprint arXiv:2509.20904},
  year={2025}
}

📄 License

This project is licensed under the Apache License 2.0.