nagbhaskar55/gemma-3-270m-bhaskar-finetune

gemma-3-270m-bhaskar-finetune

`google/gemma-3-270m-it` fully fine-tuned on a synthetic legal/financial instruction set, to answer questions grounded in a passage supplied in the prompt.

Results

Held-out split, loss over assistant tokens only:

Trained on L4 x1 in 37 minutes (10,520 tok/s, peak 20.29 GB).

Versus a 125M model trained from scratch on the same data

The same dataset was used to fine-tune nagbhaskar55/slm125mlive-base, a 125M model pretrained from scratch on 2.04B tokens. Per-token perplexity is not comparable across a 16k and a 262k vocabulary, so the like-for-like measure is bits per character of the reference answers:

A 19% reduction. Both were fine-tuned on identical data, so the gap reflects pretraining scale rather than fine-tuning.

Training data

7,960 curated synthetic instruction pairs generated from US case law, SEC filings and educational web text, with gemini-3.6-flash as teacher and gemini-3.1-flash-lite as an LLM judge (grounding / correctness / instruction-following, each >= 4 of 5), then exact + n-gram + embedding deduplication and 13-gram decontamination against CaseHOLD.

• —Tasks: {'summarization': 1600, 'extraction': 1600, 'grounded_qa': 3200, 'rewriting': 1560}
• —Sources: {'sec': 3186, 'case-law': 3158, 'fineweb-edu': 1616}
• —Refusals: 423 items answer exactly "Not stated in the context."
• —Re-tokenized for Gemma: 7,617 train examples, 4,896,142 tokens, 8.7% carrying loss. 105 examples exceeded 1,024 tokens and were dropped rather than truncated.

Prompt format

Gemma's own chat template. It has no separate system channel — a system turn is folded into the first user turn, which is how this model was trained:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("nagbhaskar55/gemma-3-270m-bhaskar-finetune")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/gemma-3-270m-bhaskar-finetune")

system = "You are a legal and financial assistant. Use only the provided context."
messages = [{"role": "user", "content": f"{system}\n\n{passage}\n\n{question}"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

out = model.generate(
    ids, max_new_tokens=200,
    # Gemma turns end with <end_of_turn> (106), not <eos> (1). Pass both, or the
    # model will not stop.
    eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<end_of_turn>")],
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Loss was applied only to the model turn (8.7% of tokens).

Limits

At 270M parameters this reads the passage you give it; it is not a knowledge base. Answers are fluent and usually well-formed, but figures and entities are still sometimes wrong, and it can miss part of a multi-part question. Training data is synthetic, from a single teacher model, so its biases carry over. Not legal or financial advice. Licensed under the Gemma Terms of Use, which carry use restrictions and must be passed downstream.