strykes/tiny-giant-2500-q4_k_m

Tiny-Giant 2500 — Q4KM (pilot)

Full fine-tune of Qwen2.5-Coder-1.5B-Instruct on the Tiny-Giant pilot dataset (~2,508 execution-verified agentic coding samples), quantized to Q4_K_M for llama.cpp deployment.

This is the pilot (`TAG=2500`) checkpoint — it validates the train → quant → eval pipeline, not the final production model (target ≥30k samples).

Model file

Training summary

Usage (llama.cpp)

Pin the ChatML template explicitly. This model was trained with a custom Hermes/ChatML renderer (train_tiny_giant.render_conversation); do not rely on template auto-detection.

# llama-server
llama-server -m tiny-giant-2500-Q4_K_M.gguf --chat-template chatml --ctx-size 4096

# llama-cpp-python server
python -m llama_cpp.server --model tiny-giant-2500-Q4_K_M.gguf \
  --chat_format chatml --n_ctx 4096 --port 8088

from llama_cpp import Llama

llm = Llama(model_path="tiny-giant-2500-Q4_K_M.gguf", n_ctx=4096, n_gpu_layers=-1)
out = llm.create_completion(
    prompt="<|im_start|>user\nWrite a Python function to merge two sorted lists.\n<|im_start|>assistant\n",
    max_tokens=512,
    temperature=0.0,
    stop=["<|im_start|>"],
)
print(out["choices"][0]["text"])

Evaluation

Benchmark and internal eval results are populated after the Vast GPU eval run completes. See the companion comparison card in the source repository (MODEL_CARD-2500.md).

Limitations

• —Pilot scale (2.5k samples): format adherence and debugging habits should improve vs base; absolute codegen benchmarks may move little until the full 30k run.
• —Q4_K_M quantization: if eval shows >~4pt drop vs bf16, re-quantize the kept f16 GGUF to Q5KM — no retraining needed.
• —Tool-use eval requires prompts built with the same renderer used in training.

Source

Training pipeline and dataset factory: Tiny-Giant (local repo).

License

Apache-2.0 (inherits from Qwen2.5-Coder-1.5B-Instruct base).