HIMANSHUKUMARJHA/minicpm5-1b-code-lora

HIMANSHUKUMARJHA/minicpm5-1b-code-lora

LoRA adapter for `openbmb/MiniCPM5-1B`, fine-tuned for coding assistance.

Trained by AutoTune, a scheduled fine-tuning pipeline on Modal that only publishes an adapter when it beats the previous best on a held-out slice.

Results

The eval gate published this version only because it beat the stored previous best on the same fixed holdout.

Caveat on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal rather than a capability benchmark; pass@1 on HumanEval or MBPP would be the meaningful measure and is not yet wired up.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "HIMANSHUKUMARJHA/minicpm5-1b-code-lora")

messages = [{'role': 'user', 'content': 'Write a Python function that reverses a linked list.'}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
inputs.pop("token_type_ids", None)   # this architecture's generate() rejects it
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers>=4.51, which is the first version that reads MiniCPM's standalone chat_template.jinja.

Training

Reproduced with AutoTune:

modal run finetune.py --profile code

Data handling

The dataset is shuffled with a fixed seed, then a 200-row holdout is taken before training. Rows the formatter cannot parse are dropped and exact duplicates removed, because these datasets contain repeats that would otherwise leak holdout examples into training.

12,515 exact duplicates were removed from this dataset before training.

Note on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal, not a capability benchmark. Pass@1 on HumanEval or MBPP would be the meaningful measure.

Limitations

This is a 1B parameter model. It is useful for coding assistance at small scale and for on-device or cost-sensitive settings, but it will not match a large general model. Outputs should be validated before use. The adapter inherits any bias present in the training dataset.