ceselder/personahub-lora-oracles

PersonaHub LoRA Oracles

25,000 rank-1 LoRA weight diffs trained on PersonaHub personas on Gemma 3 4B-it. Designed for training weight oracle models that can read LoRA weights and describe model behavior.

Dataset Structure

• —personas.parquet — 25K persona descriptions (personaid, personatext)
• —dataset_config.json — Full reproduction config (model, hyperparams, prompts)
• —loras/ — Raw LoRA weight files (.pt), one per persona (~3.75MB each, fp16)
• —direction_tokens/ — Precomputed projected direction vectors for oracle training (~2.5MB each)

How LoRAs Were Trained

Each persona was converted to a system prompt: "You are the following persona. Stay in character at all times.

{persona}"

50 rollouts were generated per persona using Gemma 3 4B-it with diverse user messages. Each LoRA was trained for 32 steps using batched MultiTaskLoRALinear (8 LoRAs simultaneously via bmm).

Direction Tokens

For each LoRA, we extract reading and writing direction vectors from each module at each layer:

• —Reading direction: row of A matrix (which residual stream features activate this component)
• —Writing direction: column of B matrix, projected through base model matrices to R^{d_model}

This gives 476 vectors per LoRA (34 layers × 7 modules × 1 rank × 2 directions), each in R^2560.

Base Model

google/gemma-3-4b-it

Config

• —Rank: 1, Alpha: 2
• —32 training steps, LR 3e-4, AdamW
• —Target modules: q/k/v/oproj, gate/up/downproj (language model only, vision tower excluded)
• —Source: PersonaHub persona split (first 25K)