Caglana/qwen0.5b-tinylora-ds-assistant

Qwen2.5-0.5B — Four Parameter-Efficient Fine-Tunes — Data Science Assistant

This repo hosts four separate fine-tunes of Qwen/Qwen2.5-0.5B-Instruct on the same synthetic data-science-assistant corpus, each exploring a different way to adapt a small number of parameters: TinyLoRA (a shared-projection adapter spread across every layer), a standard LoRA confined to one layer, and two flavors of growing new transformer blocks on top of the frozen base. They are four independent checkpoints, not four pieces of one combined model — pick the revision for the method you want.

main currently holds TinyLoRA (tinylora-v1); the other three are reachable only by their own git tag, since main can only show one method's weight files at a time (uploading a new checkpoint here never deletes an older, different-shaped one's files — see the note under each section's loading snippet).

Trained with the llm_with_tiny_lora project.

Loading a specific version

The two PEFT adapters (TinyLoRA, Layer-LoRA) share this setup -- pick the revision for the one you want:

pip install git+https://github.com/huggingface/peft.git  # TinyLoRA needs peft's main branch
pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_id = "Caglana/qwen0.5b-tinylora-ds-assistant"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)  # same tokenizer files on every revision
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype="auto")

model = PeftModel.from_pretrained(base_model, adapter_id)  # main = TinyLoRA (tinylora-v1)
# model = PeftModel.from_pretrained(base_model, adapter_id, revision="layer-lora-v1")  # or this one

layerexpand/layergrow are not PEFT adapters -- they are whole merged models with a non-standard (wrapped) architecture, loadable only through this project's own layer_expand/ layer_grow packages (see each of those sections' own loading snippet below); plain AutoModelForCausalLM.from_pretrained will not reconstruct them correctly.

What's new in this version

Retrained on an expanded synthetic corpus (see below for what's new in this version).

Training data

A synthetic data-science assistant corpus generated by the same project: concept Q&A, runnable pandas/matplotlib/scikit-learn/statistics/SQL tasks, and tool-use conversations in which the assistant calls a data tool and answers from what it returns. The full corpus is ~4M records (~11 GB across 40 shards); each method below trained on its own subset of it (see that method's own hyperparameter table).

New in this dataset version: tool-use conversations

Earlier versions taught the assistant what to say. This version adds a fourth generator, `data_generator_tool_use_call.py`, which teaches it what to do when it has tools: a workspace of registered datasets it can list, describe, profile, query with SQL or Python and cross-validate a baseline on, plus the knowledge base the rest of the corpus is generated from. See the Tool use section for the format and the tool catalogue.

Tool arguments and schemas are real — dataset keys, column names, targets, positive labels and split structure come from the same Kaggle catalogue the code corpus uses. Tool results are simulated, under one rule: catalogue facts (row counts, positive rates, schemas) repeat across records, while everything else (missing counts, means, fold scores) is drawn fresh per conversation. A value that changes every time it appears cannot be memorised, so the only way for the loss to fall on the final answer is to copy it out of the tool result above — which is the behaviour being taught.

Also in this corpus: a Kaggle-grounded code corpus

The earlier versions trained on two generators:

• —Knowledge-base Q&A (kb_questions.py) — hand-written data-science concepts turned into nine question shapes per concept (definition, practice, tradeoffs, pitfalls, checks, overview, review, decision, failure map), so the same material is asked for in many different ways.
• —Synthetic code tasks (code_tasks.py) — 45 pandas / matplotlib / scikit-learn / statistics / SQL tasks, each rendered against many invented domains and against a Technique (estimator, scaling, CV scheme, search space, metric, all moving together).

A third generator, `data_generator_code_base.py`, is the same code-task idea anchored to datasets that actually exist. 24 open Kaggle problems — Titanic, House Prices, Spaceship Titanic, Credit Card Fraud, Telco Churn, Adult Census Income, Pima Diabetes, Heart Failure, Home Credit Default Risk, Santander, Porto Seguro, IEEE-CIS Fraud, Bike Sharing, NYC Taxi Duration, Store Sales, Rossmann, Otto, MNIST, Give Me Some Credit, Wine Quality, Mobile Price, NYC Airbnb, Cardiovascular Disease, Mercedes-Benz Greener Manufacturing — each written against the five stages of a modelling project:

Two properties are enforced across every emitted answer, and they are what the fine-tune is meant to absorb:

• —Generic — the dataset-specific part is a short header (DATA, TARGET, DROP, domain features); everything below it is written against dtypes and column selectors, so the same body transfers to any tabular problem instead of being Titanic-shaped.
• —Reliable — every fitted step lives inside a Pipeline fit inside the CV loop, the split respects the group/time structure the dataset actually has, leakage columns are dropped by name, seeds are pinned, and the specific failure being avoided is named on a Watch out: line rather than left implicit.

Every answer in both code generators has the same three-part shape — a lead sentence saying what the code does and why, one runnable block, and a Watch out: line carrying the knowledge-base concept it is grounded in — so the corpora teach one output format rather than three.

TinyLoRA (tinylora-v1)

Unlike standard LoRA, which trains a pair of low-rank matrices A/B per target module, TinyLoRA freezes a set of random projection matrices and trains only a small weighting vector v on top of them:

ΔW = Σᵢ vᵢ Pᵢ

The adapter is spread across every transformer layer, but each module contributes only u trained scalars rather than a full low-rank pair.

This checkpoint trains 21,504 parameters — roughly 0.004% of the ~494M-parameter base model.

Loading this version:

model = PeftModel.from_pretrained(base_model, "Caglana/qwen0.5b-tinylora-ds-assistant", revision="tinylora-v1")
# PEFT adapter -- loads with the snippet in "Loading a specific version" above.

Hyperparameters

Full config: `configs/sft_ds_assistant.yaml`.

Results

Checkpoint at step 12000 (epoch ≈7.68) reached eval_loss 1.7902 (perplexity ≈5.99) on the held-out split during training.

No scored base-vs-adapter benchmark was found for this checkpoint in outputs/eval_results.json, so only the training-time loss is reported here. Run bash eval.sh against this checkpoint to fill in the full comparison.

Layer-LoRA (layer-lora-v1)

This is a layer-scoped LoRA: a standard LoRA update

ΔW = (α / r) · B A

installed on the attention projections of a chosen subset of transformer layers, with every other layer left at its base weights. The late layers carry the most task-specific representation, so an adapter placed there moves output style and format the most per trained parameter — and confining it to those layers keeps the rest of the network's behaviour exactly as the base model shipped it.

This checkpoint trains 30,720 parameters — roughly 0.006% of the ~494M-parameter base model.

Loading this version:

model = PeftModel.from_pretrained(base_model, "Caglana/qwen0.5b-tinylora-ds-assistant", revision="layer-lora-v1")
# PEFT adapter -- loads with the snippet in "Loading a specific version" above.

Hyperparameters

Full config: `configs/sft_layer_lora.yaml`.

Results

Checkpoint at step 16000 (epoch ≈10.24) reached eval_loss 0.5837 (perplexity ≈1.79) on the held-out split during training.

No scored base-vs-adapter benchmark was found for this checkpoint in outputs/eval_results.json, so only the training-time loss is reported here. Run bash eval.sh against this checkpoint to fill in the full comparison.

layer_expand (layer-expand-v1)

Appends one new transformer block on top of the frozen base model and trains only that block -- the base's own weights (optionally themselves built by merging one or more finished adapters first, see "Frozen base built from" below) never change.

This checkpoint trains 46,207,744 parameters — roughly 9.354% of the ~494M-parameter base model.

Loading this version:

# Not a PEFT adapter -- a whole merged model with a non-standard (wrapped) block.
# Needs this project's own layer_expand package to rebuild the architecture:
from layer_expand.model import load_expanded_model
model = load_expanded_model("Caglana/qwen0.5b-tinylora-ds-assistant", revision="layer-expand-v1")

Hyperparameters

Full config: `configs/sft_layer_expand.yaml`.

Results

Checkpoint at step 4250 (epoch ≈2.72), scored against the un-adapted base model on the same held-out split (data/synthetic/dataset/sft_eval.jsonl):

The headline number is perplexity: 15.57 → 1.06, a 93% cut, from an adapter that trains a fraction of a percent of the base model's weights.

Reported as measured, including what got worse: valid-Python rate 6.7% → 6.7%. Generation-side metrics are scored on a small sample of prompts with greedy decoding, so single-percent moves there are noise; a swing as large as the valid-Python rate's is not, and is the thing to watch on the next run.

Note on comparability. Later checkpoint of the same run as README's 'layerexpand checkpoint-11200' (run 7, not itself recorded here since it was only ever captured as a printed eval.sh table) -- same outputs/sft-layer-expand-wide run, same configs/sftlayerexpand.yaml (maxsteps 20000), 4800 more steps. evalloss/perplexity are essentially flat versus checkpoint-11200 (0.0484/1.0496 -> 0.0617/1.0636) despite the extra training, and rougelf1/tokenf1 fell (0.6948/0.7014 -> 0.4933/0.5394) even though the base row's own evalloss/perplexity barely moved (2.7433/15.5388 -> 2.7452/15.5682); its rougelf1/tokenf1 did move somewhat (0.0974/0.2323 -> 0.0867/0.2094), most likely from a different random subset of generated examples rather than a split change, since the split-independent loss numbers held steady. Read together: this checkpoint is not an improvement over checkpoint-11200 -- flat teacher-forced loss plus falling free-generation metrics after more training is the signature of a run that has already overfit past its useful point, on top of the leakage/train-eval-overlap check already flagged for checkpoint-11200 given its own near-1.0 perplexity. codevalidrate and the newly-added repetitionlooprate both tie exactly with their own base row here, so neither is informative at this sample size.

layer_grow (layer-grow-v1)

Grows the stack by one or more further blocks on top of an already-expanded model, keeping every previously-grown block trainable too (not just the newest one) -- see "Round" rows below for how many rounds this checkpoint has been through.

This checkpoint trains 403,266,816 parameters — roughly 81.633% of the ~494M-parameter base model.

Loading this version:

# Not a PEFT adapter -- a whole merged model, one or more rounds grown.
# Needs this project's own layer_grow package to rebuild the architecture:
from layer_grow.model import load_grown_model
model = load_grown_model("Caglana/qwen0.5b-tinylora-ds-assistant", revision="layer-grow-v1")

Hyperparameters

Full config: `configs/sft_layer.yaml`.

Results

Checkpoint at step 1750 (epoch ≈1.12) reached eval_loss 0.1034 (perplexity ≈1.11) on the held-out split during training.

No scored base-vs-adapter benchmark was found for this checkpoint in outputs/eval_results.json, so only the training-time loss is reported here. Run bash eval.sh against this checkpoint to fill in the full comparison.

Tool use

This version is trained to call tools in Qwen2.5's native tool format, the one the base model's chat template already defines: the offered tools arrive in the system prompt inside <tools> tags, and the model answers with one <tool_call> block per call.

<tool_call>
{"name": "describe_dataset", "arguments": {"dataset": "titanic"}}
</tool_call>

Pass the tools through the chat template and the model sees exactly what it was trained on. Parse the <tool_call> blocks out of its reply, run them yourself, and hand the results back as tool messages.

Calling it

Continuing from the snippet above:

import json, re

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "describe_dataset",
            "description": (
                "Profile a registered dataset: rows, columns by type, the target and its metric, "
                "split structure and known data-quality issues."
            ),
            "parameters": {
                "type": "object",
                "properties": {"dataset": {"type": "string", "description": "Dataset key."}},
                "required": ["dataset"],
            },
        },
    }
]

# The system prompt this adapter was trained under. Keep it — see the note below.
SYSTEM = (
    "You are a senior data scientist. You answer questions about data engineering, feature "
    "engineering, statistics, machine learning and visualisation, and you write working code when "
    "code is what the question calls for. Be direct and concrete: name the trade-off, name the "
    "failure mode, and say what to check. When you write code, give a short explanation, one "
    "runnable block, and a note on what usually goes wrong."
)

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Profile the titanic dataset for me."},
]


def generate(messages):
    text = tokenizer.apply_chat_template(
        messages, tools=TOOLS, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()


reply = generate(messages)
calls = [json.loads(block) for block in re.findall(r"<tool_call>\s*(.*?)\s*</tool_call>", reply, re.S)]

# Run each call yourself, then give the results back and generate again.
for call in calls:
    result = run_my_tool(call["name"], call["arguments"])  # your implementation
    messages += [
        {"role": "assistant", "content": reply},
        {"role": "tool", "content": json.dumps(result)},
    ]

print(generate(messages) if calls else reply)

Two details worth knowing:

• —Replay the assistant turn as content. Training wrote tool calls into message content, exactly as the chat template renders them, so replaying them that way is what the model saw. The structured form ({"role": "assistant", "tool_calls": [...]} with tools= on the template) renders to the same string byte for byte, so either works.
• —Greedy decoding (do_sample=False) is how the adapter is evaluated, and it makes the call format reproducible.

The system prompt is not optional in practice. Every tool-use conversation in training sat under the prompt above, and the adapter learned the behaviour in that context. Greedy runs of this checkpoint asked to profile a dataset called a tool in every condition tested with that prompt — one, two and four tools offered, bf16 and fp32 — and in one of twelve without it. With no system prompt it tends to ask a follow-up question ("which file?") instead of calling anything. How many tools you offer made little difference; the system prompt made all of it.

The tools it was trained on

The offered list is drawn per conversation — the tools a question needs plus one to three others, in shuffled order — so the model reads the <tools> block rather than memorising one fixed toolbox.

Your own tools are not these seven. How far the behaviour carries to a different toolbox is a question of how well a 0.5B adapter generalises — the format is what was trained, not the catalogue.

Behaviours it was trained for

9,903 conversations, of which 6,843 make at least one call (5,435 one call, 1,315 two, 93 three or four). The rest are the cases where calling a tool is the wrong move — which is half the skill:

Limitations

• —0.5B base model: capable of short, focused answers but not competitive with larger models on complex multi-step reasoning.
• —Narrow domain: tuned specifically for data-science Q&A and code generation; general-purpose chat quality is not a training objective.
• —The generated code is written to be runnable, but it is generated — read it before running it against anything that matters.
• —Identity/safety guardrails (e.g. not identifying as "Qwen"/"Alibaba") are implemented at the application layer in the project's `chat.py` wrapper, not baked into any of these checkpoints' weights.
• —layerexpand/layergrow revisions are not standard PEFT adapters or stock AutoModelForCausalLM checkpoints — they need this project's own packages to load (see each section's loading snippet above).

License

Base model (Qwen/Qwen2.5-0.5B-Instruct) is Apache 2.0. Every checkpoint in this repo is released under the same terms.