Modelpublic

Afrostnova/clip-text-msmarco-corrector

sourceHugging Faceotherupdated 4mo agoView on Hugging Face

1likes8downloads

Model Card

clip-text-msmarco-corrector

A vec2text corrector model for the CLIP ViT-L/14 text encoder embeddings (the text encoder used by Stable Diffusion v1.5, identical to `openai/clip-vit-large-patch14`).

Given an initial text hypothesis from the inversion model, this model iteratively corrects it to better match the target embedding. It is used together with the inversion model `Afrostnova/clip-text-msmarco-inversion` — it is not used standalone and it reuses the inversion model's tokenizer.

—Base architecture: vec2text CorrectorEncoderModel
—Embedder: CLIPTextModel — openai/clip-vit-large-patch14
—Training data: MS MARCO
—Built on inversion alias: ClipTextModel-msmarco__msl32__58epoch

⚠️ Requirements

Requires the vec2text fork that supports the CLIPTextModel embedder (the one this model was trained with), not the upstream pip install vec2text package.

Usage

python

import vec2text

inv = vec2text.models.InversionModel.from_pretrained("Afrostnova/clip-text-msmarco-inversion")
cor = vec2text.models.CorrectorEncoderModel.from_pretrained("Afrostnova/clip-text-msmarco-corrector")
corrector = vec2text.load_corrector(inv, cor)

text = vec2text.invert_embeddings(
    embeddings=embeddings,   # CLIP text-encoder last_hidden_state pooled at EOS
    corrector=corrector,
    num_steps=20,
    sequence_beam_width=1,
)