Modelpublic

Spiritual4/activation-oracle-gemma-3-12b-it

sourceHugging Facegemmaupdated 6mo agoView on Hugging Face

0likes9downloads

Model Card

Activation Oracle for Gemma 3 12B-IT

License Notice: This LoRA adapter is a derivative of google/gemma-3-12b-it and is subject to the Gemma Terms of Use. By using this model, you agree to those terms.

A LoRA adapter that turns Gemma 3 12B-IT into an activation oracle — an LLM that can answer natural language questions about another model's internal activations.

What is an Activation Oracle?

Activation oracles accept neural activations as an additional input modality and answer arbitrary questions about them. Instead of looking at SAE feature IDs and hoping for descriptions, you can ask:

—"What language is being generated here?"
—"What word is the model trying to produce?"
—"Is the model confident about this token?"

Based on Karvonen et al. (2025) "Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers".

Training Details

—Base model: google/gemma-3-12b-it
—Architecture: LoRA (rank 64, alpha 128, dropout 0.05)
—Target modules: All linear layers in language model (vision tower excluded)
—Training data: LatentQA + 10 classification tasks + PastLens context prediction (~1M examples)
—Training: 1 epoch, 32,021 steps, batch size 32
—Hardware: 8x NVIDIA A100 40GB (p4d.24xlarge)
—Optimizations: TF32 tensor cores, Liger Kernel fused ops
—Training time: ~8 hours
—Final loss: ~1.3-1.8

Usage

Requires the activation_oracles codebase.

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model + oracle adapter
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-12b-it", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "Spiritual4/activation-oracle-gemma-3-12b-it")

# See activation_oracles repo for full inference pipeline
# (activation capture, injection into layer 1, oracle prompting)

Acknowledgments

—Karvonen et al. for the activation oracles framework and codebase
—Google DeepMind for Gemma 3 and Gemma Scope 2

Notice

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.