tpls/gemma4-coder-tool-eval
Gemma-4 Coder — tool-calling gate suite The 8-case eval behind the tool-call pass rate in our model cards' model-index: positive prompts where the model must emit a structured tool call, plus a no-tool abstain case (it must answer directly, not hallucinate a call). Use it to measure any local tool-calling model the same way we do — or to reproduce our numbers. from datasets import load_dataset cases = load_dataset("tpls/gemma4-coder-tool-eval", split="train") # each: {name… See the full description on the dataset page: https://huggingface.co/datasets/tpls/gemma4-coder-tool-eval.
020
1---2license: apache-2.03task_categories:4 - text-generation5tags:6 - function-calling7 - tool-use8 - evaluation9 - gemma410pretty_name: Gemma-4 Coder — tool-calling gate suite11configs:12 - config_name: default13 data_files: tool_eval_cases.jsonl14---15 16# Gemma-4 Coder — tool-calling gate suite17 18The 8-case eval behind the **tool-call pass rate** in our model cards' `model-index`:19positive prompts where the model must emit a structured tool call, plus a no-tool20**abstain** case (it must answer directly, not hallucinate a call). Use it to measure21any local tool-calling model the same way we do — or to reproduce our numbers.22 23```python24from datasets import load_dataset25cases = load_dataset("tpls/gemma4-coder-tool-eval", split="train") # each: {name, user, tools, expect}26```27 28## Scoring a model29 30Serve a GGUF on `llama.cpp` (`llama-server --jinja`), send each case's `user` + `tools`31the normal OpenAI way, and check the reply against `expect`. **Two rates**, because32`llama.cpp --jinja` doesn't recognise gemma-4's native tool-call markup:33 34- **raw** — llama.cpp's native parse. For gemma-4 it *structurally undercounts* (blind to the format).35- **shim** — the same outputs re-parsed for the native markup. This is the number that reflects training.36 37The shim is a tiny serve-side post-processor — ready-to-use (drop-in litellm callback +38a standalone parser, Apache-2.0) at **[`tpls/gemma4-tool-shim`](https://huggingface.co/tpls/gemma4-tool-shim)**,39where the recovery algorithm is also documented so you can re-implement it.40 41```python42# sketch: per case, POST to your OpenAI-compatible endpoint, then43# raw_ok = response had a structured tool_call (or correctly abstained)44# shim_ok = same, after running the reply through the shim parser45# pass_rate = mean(ok over the 8 cases). expect == [] means "must NOT call a tool".46```47 48## Files49 50| File | What |51|------|------|52| `tool_eval_cases.jsonl` | one case/line: `{name, user, tools, expect}` |53| `tool_calib.txt` | imatrix calibration text (code + tool-call markup) used when quantizing |54 55## Scope & license56 57Authored by us, fully permissive (apache-2.0). This is an **eval** set, not training data —58our training mix is a derivative of public datasets and is **not** redistributed (its59reproducible recipe lives on each model card). Pairs with the60[`tpls/gemma4-tool-shim`](https://huggingface.co/tpls/gemma4-tool-shim) helper and the models below.61 62## Used by63 64- [`tpls/Huihui-gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-GGUF`](https://huggingface.co/tpls/Huihui-gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-GGUF)65- [`tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-GGUF`](https://huggingface.co/tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-GGUF)66- [`tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5`](https://huggingface.co/tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5)67- [`tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5-GGUF`](https://huggingface.co/tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5-GGUF)68