PixelProof/gemma-4-31b-pixelproof-lora-adapters

PixelProof LoRA adapters for Gemma 4 31B

LoRA adapters for `google/gemma-4-31B-it` trained with supervised fine-tuning (SFT) on visual questions generated by PixelProof, as described in the paper PixelProof: Visual Question Generation through Forward–Inverse Agreement. In PixelProof, a coding agent writes a forward program that draws images and computes their answers and an inverse program that must recover each answer from the image alone; a question is kept only if the two agree.

Each adapter is stored in its own subfolder. The repository contains no merged weights.

Training

Training uses transformers.Trainer and PEFT LoRA adapters on the language model's attention and feed-forward projections; the vision tower is not directly adapted. The main recipe uses rank 16, scaling (alpha) 32, dropout 0.05, bfloat16, one epoch, learning rate 1e-4, cosine decay, 3% warm-up, batch size two, and eight gradient-accumulation steps. The loss covers only the assistant answer and the turn-end token. Adapter names ending in -accum16 use sixteen accumulation steps.

The main training data contains only profile-steered worlds. The fixed split assigns 748 of 793 replay-verified worlds to training (27,762 training instances, with 555 reserved for validation) and holds out 45 worlds, one per model–profile campaign. Worlds, not instances, are held out, because instances from the same world share a sampler and renderer.

Adapters and paper results

<profile> is one of the nine discovery profiles: correspondence, measurement, prior_conflict, reveal_declared_transform, reveal_global_structure, search, state_tracking, topology, tracing.

Results for Gemma 4 31B reported in the paper:

• —Held-out profile-steered worlds, mean of four splits: 64.4% → 74.6% (+10.2 pp). The average over the three fine-tuned models is +10.7 pp.
• —Excluded discovery profiles, mean over nine profiles: +7.3 pp (average over the three models: +8.5 pp).
• —Transfer to 1,260 instances from 252 model-feedback-steered worlds: 30.7% → 37.8% (+7.1 pp; average over the three models: +5.3 pp).
• —16 external benchmarks, mean over four splits: 60.0 → 61.1 (+1.0 points; average over the three models: +1.9 points).
• —Direct training on model-feedback-steered worlds, 250 instances from 50 held-out worlds: +19.2 pp.

Additional adapters not reported in the paper

The following adapters come from additional runs that the paper does not report. They are released for completeness only.

• —gemma4-31b-lora-bf16
• —gemma4-31b-lora-bf16-model-feedback-variant-seed101
• —gemma4-31b-lora-bf16-model-feedback-variant-seed102
• —gemma4-31b-lora-bf16-model-feedback-variant-seed103
• —gemma4-31b-lora-bf16-profile-steered-4source
• —gemma4-31b-lora-bf16-profile-steered-variant-seed101-balanced
• —gemma4-31b-lora-bf16-profile-steered-variant-seed102-balanced
• —gemma4-31b-lora-bf16-profile-steered-variant-seed103-balanced
• —gemma4-31b-lora-bf16-spatially-steered
• —gemma4-31b-lora-bf16-spatially-steered-variant-seed101
• —gemma4-31b-lora-bf16-spatially-steered-variant-seed102
• —gemma4-31b-lora-bf16-spatially-steered-variant-seed103

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base_id = "google/gemma-4-31B-it"
repo_id = "PixelProof/gemma-4-31b-pixelproof-lora-adapters"
subfolder = "gemma4-31b-lora-bf16-profile-steered"

processor = AutoProcessor.from_pretrained(repo_id, subfolder=subfolder)
model = AutoModelForImageTextToText.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, repo_id, subfolder=subfolder)

Prompt the model with only the image, the question text, and the answer options. The evaluation data are available in `PixelProof/PixelProof-Evaluation` and `PixelProof/PixelProof-Difficulty-Feedback`.

Limitations

All PixelProof training data are synthetic two-dimensional images, and the gains may not extend to natural images. On the external benchmarks, some benchmarks show smaller gains or regressions. Agreement between the forward and inverse programs shows that an answer can be recovered from the image, but not necessarily that the question is clearly worded.

Project resources

• —PixelProof website
• —Source code