nagbhaskar55/slm125mlive-bhaskar-finetune

slm125mlive-bhaskar-finetune

Instruction-tuned from nagbhaskar55/slm125mlive-base, a 125M-parameter Llama-architecture model pretrained from scratch on US case law, SEC filings and educational web text.

What changed

Trained 3 epochs on H100x1; best checkpoint was epoch 0 of 3.

Training data

7,960 synthetic instruction pairs generated from the same corpus the base model was pretrained on, with gemini-3.6-flash as teacher and gemini-3.1-flash-lite as an LLM judge. Every answer is grounded in a passage supplied in the prompt.

• —Tasks: {'summarization': 1600, 'extraction': 1600, 'grounded_qa': 3200, 'rewriting': 1560}
• —Sources: {'sec': 3186, 'case-law': 3158, 'fineweb-edu': 1616}
• —Difficulty: {'medium': 2755, 'hard': 1134, 'easy': 4071}
• —Refusals: 423 items answer exactly "Not stated in the context." when the passage does not contain the answer
• —Lengths: mean 629.5 tokens, p90 798, max 1019 (limit 1024)

Curation: LLM judge on grounding / correctness / instruction-following (each >= 4 of 5), exact + 8-gram + embedding near-duplicate removal, and 13-gram decontamination against CaseHOLD. 10,958 raw pairs were reduced to 7,960.

Prompt format

Loss was applied to assistant tokens only. The chat template ships in tokenizer_config.json:

<|bos|><|system|>{system}<|eos|><|user|>{user}<|eos|><|assistant|>{answer}<|eos|>

System prompt used in training:

You are a legal and financial assistant. Use only the provided context.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("nagbhaskar55/slm125mlive-bhaskar-finetune")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/slm125mlive-bhaskar-finetune")

messages = [
    {"role": "system", "content": "You are a legal and financial assistant. Use only the provided context."},
    {"role": "user", "content": "<passage>\n\n<your question>"},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=200)[0][ids.shape[1]:],
                 skip_special_tokens=True))

Intended use and limits

This model reads a passage you supply and answers from it. It is not a knowledge base: at 125M parameters, closed-book answers are unreliable, and the training data deliberately teaches grounded reading rather than recall.

It was tuned on synthetic data from a single teacher model, so it inherits that teacher's biases and error modes. Do not use it for legal or financial advice. CaseHOLD was held out of both pretraining and fine-tuning; the base model scored below chance zero-shot there, and this checkpoint has not been re-evaluated on it.