myyycroft/Gemma-4-E2B-AmbigQA-full-short-form-prompt-member-3
044
myyycroft/Gemma-4-E2B-AmbigQA-full-short-form-prompt-member-3
Ensemble member 3 (seed 3069) from run gemma4_e2b_full_small_lr_short_form_prompt.
Fine-tuned from `google/gemma-4-E2B-it` on AmbigQA.
Evaluation note
Metrics below are computed on small fixed subsets, not full benchmarks. Exact subset sizes: AmbigQA (128), IFEval (64), MMLU (228). Reported values are from the final training step (step=1620, epoch=3).
Dataset
- Hub dataset: `sewon/ambig_qa` (
config_name=light) - Revision:
6e667596df70f17ba3c8e7be4b7361f6be8b60f8 - Splits: train=
train, validation=validation - Dev holdout:
dev_fraction=0.1,split_seed=1729 - Bootstrap resampling:
False
System prompt
You answer open-domain factoid questions with a short-form answer only.
Reply with just the answer string: typically a name, place, organization, title, date, number, or brief factual phrase (often about 1–4 words). Do not write a sentence, explanation, or preamble.
Examples of the expected style:
- Who painted The Persistence of Memory? → Salvador Dalí
- When was the first iPhone released in the United States? → June 29, 2007
- What is the capital of New Zealand? → Wellington
- Who wrote Pride and Prejudice? → Jane Austen
- What is the chemical symbol for gold? → Au
- Who directed the 1994 film Pulp Fiction? → Quentin Tarantino
Formatting rules:
- Output the answer alone on a single line.
- Do not restate the question, add reasoning, markdown, bullets, or quotation marks around the answer.
- Prefer a canonical short form over a descriptive sentence.
- Give one answer only; do not list alternatives or hedge.
- Use natural short forms for dates and names (e.g. June 29, 2007; Jane Austen).Hyperparameters
Exact resolved settings used for this run (also attached as resolved_config.yaml):
model:
name: google/gemma-4-E2B-it
revision: main
dtype: bfloat16
enable_thinking: false
dataset:
name: sewon/ambig_qa
config_name: light
revision: 6e667596df70f17ba3c8e7be4b7361f6be8b60f8
train_split: train
validation_split: validation
dev_fraction: 0.1
split_seed: 1729
bootstrap: false
max_train_examples: null
max_dev_examples: null
system_prompt_set: true
adaptation:
method: full
training:
num_train_epochs: 3.0
max_steps: -1
learning_rate: 2.0e-05
weight_decay: 0.1
warmup_ratio: 0.03
per_device_train_batch_size: 4
per_device_eval_batch_size: 16
gradient_accumulation_steps: 8
max_seq_length: 512
logging_steps: 10
save_every_steps: null
save_total_limit: null
gradient_checkpointing: true
bf16: true
tf32: true
max_grad_norm: 1.0
dataloader_num_workers: 2
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1.0e-08
ensemble:
size: 5
base_seed: 42
this_member_index: 3
this_member_seed: 3069
evaluation_subsets:
ambigqa:
subset_size: 128
seed: 1001
ifeval:
subset_size: 64
seed: 1002
mmlu:
subset_size: 228
seed: 1004Metrics breakdown
The metrics in this section use small fixed subsets, not full benchmarks: AmbigQA (128), IFEval (64), MMLU (228).
Final-step ensemble mean ± std (n=5)
- AmbigQA (128) accuracy: 0.1047 ± 0.0089
- AmbigQA (128) AlignScore: 0.1760 ± 0.0073
- IFEval (64) prompt_level_strict_accuracy: 0.7312 ± 0.0339
- MMLU (228) accuracy: 0.5991 ± 0.0362
- train_loss: 1.5460 ± 0.0033
- steps: 1620
- epochs: 3
Per-member (final step)
Files
- Model weights / adapter files from
members/member_003/final/ resolved_config.yaml— full resolved training configensemble_metrics.png— train/eval curves for the whole ensemblerun_artifacts/— ensemble-level manifests, status, and captured environmentrun_artifacts/members/member_003/— this member's manifests, metadata, and statusrun_artifacts/members/member_003/predictions/— per-dataset JSONL predictions from every intermediate evaluation step
