Modelpublic

victor/functiongemma-270m-agent-sft-lora

sourceHugging Facegemmaupdated 24d agoView on Hugging Face

0likes35downloads

Model Card

FunctionGemma-270M LoRA — agentic tool-calling

LoRA adapter (r=16, alpha=32) fine-tuned from `unsloth/functiongemma-270m-it` on `victor/functiongemma-agent-sft` (7,500 agentic tool-calling examples), for generating valid FunctionGemma-style tool calls (read_file, write_file, edit_file, glob, bash).

Loss is computed over the model (assistant) turns only — the developer turn, tool declarations, user prompts, and tool responses are excluded from the loss so the adapter learns to emit correct calls rather than memorize context.

Load

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "unsloth/functiongemma-270m-it"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "victor/functiongemma-270m-agent-sft-lora")

Training

Setting	Value
Base model	unsloth/functiongemma-270m-it (270M)
Method	LoRA r=16, α=32, dropout 0 (Q/K/V/O + gate/up/down)
Loss mask	assistant turns only (labels=-100 elsewhere)
Data	train 90% / eval 10% (seed 42), max_length 8192
Optimizer	AdamW, lr 5e-5, cosine, warmup ratio 0.03
Steps	3 epochs, per-device batch 8, grad accum 4 (eff. 32)
Precision / HW	bf16, 1× A10G-small (≈24 min, ≈$0.40)

Results (held-out 750 examples, assistant-turn tokens only)

Model	Eval loss	Eval token accuracy
Base (`functiongemma-270m-it`)	4.03	0.619
This adapter	0.0037	0.9982

Eval split is held out from the same synthetic dataset, so these numbers show fit to the target distribution, not zero-shot generalization to unseen tool scenarios.