tpls/gemma4-coder-tool-eval
Gemma-4 Coder — tool-calling gate suite The 8-case eval behind the tool-call pass rate in our model cards' model-index: positive prompts where the model must emit a structured tool call, plus a no-tool abstain case (it must answer directly, not hallucinate a call). Use it to measure any local tool-calling model the same way we do — or to reproduce our numbers. from datasets import load_dataset cases = load_dataset("tpls/gemma4-coder-tool-eval", split="train") # each: {name… See the full description on the dataset page: https://huggingface.co/datasets/tpls/gemma4-coder-tool-eval.
Gemma-4 Coder — tool-calling gate suite
The 8-case eval behind the tool-call pass rate in our model cards' model-index: positive prompts where the model must emit a structured tool call, plus a no-tool abstain case (it must answer directly, not hallucinate a call). Use it to measure any local tool-calling model the same way we do — or to reproduce our numbers.
from datasets import load_dataset
cases = load_dataset("tpls/gemma4-coder-tool-eval", split="train") # each: {name, user, tools, expect}Scoring a model
Serve a GGUF on llama.cpp (llama-server --jinja), send each case's user + tools the normal OpenAI way, and check the reply against expect. Two rates, because llama.cpp --jinja doesn't recognise gemma-4's native tool-call markup:
- raw — llama.cpp's native parse. For gemma-4 it structurally undercounts (blind to the format).
- shim — the same outputs re-parsed for the native markup. This is the number that reflects training.
The shim is a tiny serve-side post-processor — ready-to-use (drop-in litellm callback + a standalone parser, Apache-2.0) at [`tpls/gemma4-tool-shim`](https://huggingface.co/tpls/gemma4-tool-shim), where the recovery algorithm is also documented so you can re-implement it.
# sketch: per case, POST to your OpenAI-compatible endpoint, then
# raw_ok = response had a structured tool_call (or correctly abstained)
# shim_ok = same, after running the reply through the shim parser
# pass_rate = mean(ok over the 8 cases). expect == [] means "must NOT call a tool".Files
Scope & license
Authored by us, fully permissive (apache-2.0). This is an eval set, not training data — our training mix is a derivative of public datasets and is not redistributed (its reproducible recipe lives on each model card). Pairs with the `tpls/gemma4-tool-shim` helper and the models below.