mihretgold/Steering_Vision_Language_Models_with_Subjective_Concepts

Steering Vision Models with Subjective Concepts

Exploring whether natural-language feedback can adapt a frozen vision-language model (CLIP) to user-defined visual concepts at inference time — without fine-tuning.

Problem

CLIP works well for generic queries ("a dog on a beach") but struggles with subjective, user-specific concepts ("a person looking guilty", "my dog looking guilty"). Traditional fine-tuning is expensive and requires labeled data.

Our approach: Instead of updating model weights, update the query embedding using natural language feedback with a progressive multi-stage pipeline.

Progressive Pipeline Architecture

This demo implements a 6-stage progressive steering pipeline:

Stage 1: Baseline CLIP          → Pure retrieval (q)
Stage 2: LLM Feedback           → Linear steering (q' = q + α·Σw·p - β·Σw·n)
Stage 3: Contrastive Subspace   → Centroid-based steering
Stage 4: Energy-Based           → Gradient descent optimization
Stage 5: Per-Concept Weighted   → Normalized per-attribute weights
Stage 6: SAE PRF Steering       → Pseudo-relevance feedback in SAE latent space

Each stage builds on the previous, with per-attribute weights from an LLM:

• —Weight (0–1): How much each attribute should influence steering
• —Rationale: Why the LLM chose this attribute (transparency)

How to Use This Demo

1. 1.Enter a text query (e.g., "a guilty dog", "a person looking tired")
2. 2.Optionally select a dataset (Flickr, Stanford Dogs, CelebA)
3. 3.Click Run Progressive Comparison — the LLM auto-generates positive/negative attributes
4. 4.All 6 steering stages appear side-by-side so you can compare retrieval quality
5. 5.Use + / ✕ to add or remove attributes, drag sliders to adjust weights, then re-run

Datasets

Experiment Results (ViT-B/32 baseline — for reference)

Key insight: Subjective queries benefit most from feedback — since there's no ground truth for concepts like "guilty," the model can't learn them without user guidance. Our steering approach adapts retrieval to individual user preferences at inference time.

Steering Methods Explained

Stage 1: Baseline CLIP

Pure retrieval without any steering. Computes cosine similarity between the query embedding and all image embeddings.

Stage 2: LLM Feedback (Linear Steering)

The LLM generates attributes with weights. We steer the query by adding weighted positive concept embeddings and subtracting weighted negative ones.

q' = q + α · Σ(w_i · embed(positive_i)) - β · Σ(w_j · embed(negative_j))

Stage 3: Contrastive Subspace

Computes the centroid of all positive and negative embeddings, then steers along the direction between them.

direction = normalize(mean(positive) - mean(negative))
q' = q + α · direction

Stage 4: Energy-Based Steering

Iteratively moves the query embedding via gradient descent to minimise an energy function that attracts toward positive concepts and repels from negative ones.

E(q') = -Σ sim(q', pos_i) + Σ sim(q', neg_i) + λ·‖q' - q‖²

Stage 5: Per-Concept Weighted Energy Steering

Same as Stage 4, but each attribute's influence is scaled by its LLM-assigned weight, then normalised.

Stage 6: SAE PRF Steering (Sparse Autoencoder + Pseudo-Relevance Feedback)

Uses a pretrained Multiscale Sparse Autoencoder (MSAE) with 4,096 latent features to steer in a disentangled latent space. The process:

1. 1.Encode the query and baseline top-K images through the SAE (512d CLIP space → 4,096d latent space)
2. 2.Compute mean feature residuals from the pseudo-positive images (baseline results)
3. 3.Select the top features by residual magnitude and nudge the query latents along those directions
4. 4.Decode back to CLIP embedding space and retrieve

This method is data-anchored: it uses the actual dataset neighbourhood rather than keyword matching, making it more robust for subjective queries.

q_lat = SAE.encode(query_emb)                    # 512d → 4096d
top_lat = SAE.encode(baseline_top_k_embs)         # K images
residual = mean(top_lat - mean_activations)       # feature direction
q_lat[top_features] += scale * residual[top_features]
steered = SAE.decode(q_lat)                       # 4096d → 512d

Summary

Human-in-the-Loop Design

• —Transparency: Each attribute includes a rationale from the LLM
• —Control: Users can edit attributes and weights via the UI
• —Accountability: Weights indicate LLM confidence
• —Iterative Refinement: Results improve with feedback loops

Setup (HF Spaces)

Set your GROQ_API_KEY as a Space Secret in Settings → Repository secrets. Without it, the app uses fallback attributes instead of LLM-generated ones.

Tech Stack

• —UI: Gradio (Blocks API)
• —Vision-Language Model: OpenAI CLIP ViT-B/16 [1] + PyTorch
• —Sparse Autoencoder: Pretrained MSAE (4,096 latents, TopK-64 ReLU) from WolodjaZ/MSAE [11]
• —LLM: Groq API (Llama 3.3 70B Versatile)
• —Hosting: Hugging Face Spaces (CPU)

References

[1] A. Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," ICML, 2021. arXiv:2103.00020

[7] A. JN, "Flickr8k Dataset," Kaggle, 2020. Link

[8] Z. Liu et al., "Deep Learning Face Attributes in the Wild," ICCV, 2015. CelebA

[9] A. Khosla et al., "Novel Dataset for Fine-Grained Image Categorization: Stanford Dogs," CVPR Workshop FGVC, 2011. Link

[10] HuggingFace, "CLIP Documentation," 2024. Link

[11] WolodjaZ, "MSAE — Multiscale Sparse Autoencoders," HuggingFace Hub, 2024. Link