saintsauce/unified-vlm-steering-liquid

Liquid (FoundationVision LiquidV17B): activation steering sweeps

Steered text and image generations from Liquid (FoundationVision LiquidV17B), one of the unified vision-language models in the unified-vlm-steering project. Steering adds alpha * v_hat (the per-layer unit difference-of-means vector) to the residual stream at every layer of a layer config.

Gemma-7B backbone, 28 decoder layers (0-indexed). Images are VQGAN codes (512 px), CFG 7.0, top-k 4096, top-p 0.96, temperature 0.99.

Sweeps

Every quadrant covers 7 concepts x 7 layer configs x 11 alphas x 20 prompts. Layer configs (0-indexed): L7 7, L14 14, L21 21, early 5-8, mid 12-15, late 19-22, all 0-27.

• —Quadrant names are <vector source>2<output>: txt2txt and img2txt steer text generation, img2img and txt2img steer image generation.
• —Text: the prompt in prompts/text_prompts.json (describe a day in the city, must include the concept's subject), no system prompt, greedy decoding, 96 new tokens; the subject phrase and every generated token are steered.
• —Images: the long descriptive prompts in prompts/image_prompts.json; only generated image tokens are steered.
• —alpha = 0 rows are the unsteered baselines (shared across configs).
• —Empty text outputs (degenerate generations at strong alpha): 14.
• —The minimal pairs behind the vectors: https://huggingface.co/datasets/saintsauce/unified-vlm-steering-minimal-pairs

Judging

Each record is one pairwise judgment. The judge saw the steered sample and its own alpha = 0 baseline (order randomized per record: steered_is_a says whether the steered one was shown as A) and was asked which of the two shows more of asked_pole. To cancel judge and position bias the asked pole is also randomized: in half the records it is the steered pole (asked_pole == steer_pole), where the steered sample is the expected winner, and in the other half it is the opposite pole, where the BASELINE is the expected winner. verdict is the raw A/B/Tie answer. label is D when the judge named the expected candidate (steering moved the sample toward its intended pole), N when it named the other one, and T for a tie. concept-rate_scores.csv has correct = 1 exactly when label == D. The judge saw few-shot exemplars first (a 10-image grid for images, 10 worked examples for text) and returned JSON with a short reason; reasoning is the model's thought summary. Against human labels the judge reaches Cohen's kappa = 0.62: 320 held-out pairs (none from these sweeps), each rated by three reviewers under the same blind A/B protocol, majority vote per pair. Reference-based scores of the same outputs (DreamSim pole score for images, AxBench C/I/F for text) are in scores/.

Layout

manifest.json                        exact run configs, sources, coverage, code commit
prompts/{text,image}_prompts.json
vectors/<concept>/{txt,img}.pt       steering vectors (difference of means over 100 minimal pairs)
generations/sweep_text.csv           concept, quadrant, config, alpha, prompt_idx, prompt, output
generations/sweep_image.csv          same keys + image_path (relative to generations/)
generations/images/<concept>/*.png
judge/verdicts_{text,image}.jsonl  one pairwise verdict per steered sample, with the judge's reasoning
judge/concept-rate_scores.csv      concept, quadrant, config, alpha, prompt_idx, verdict, correct
judge/summary.json                 label counts and concept rate per medium and quadrant
scores/dreamsim_scores.csv         DreamSim pole score per steered image (see scores/README.md)
scores/axbench_scores.csv          AxBench C / I / F / H per text output (see scores/README.md)