abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora

Llama-3.1-Nemotron-Nano-8B — xlam single tool-calling LoRA

Built with Llama.

A QLoRA adapter for nvidia/Llama-3.1-Nemotron-Nano-8B-v1 that turns free-form user requests plus a set of JSON tool schemas into exactly one schema-valid JSON tool call — no prose, no markdown fences, no explanations.

Trained and evaluated end to end with sommelier, a reference pipeline for reproducible tool-calling fine-tuning (deterministic data preparation, prompt digests, completion-only loss, deterministic evaluation, and a digest-gated base-vs-adapter comparison). This repository contains the adapter weights, tokenizer metadata, and the machine-readable evaluation evidence for the exact run that produced them (nemotron-8b-full-3).

Evaluation

Base model vs. this adapter on the 1,000-example held-out test split of abdelstark/sommelier-xlam-single-call-splits. Both evaluations used byte-identical prompts (prompt-set digest a0da8fa2…), greedy decoding (temperature 0.0, max_new_tokens 512), and the same conservative parser (sommelier.parser.v1) that counts every parse failure as a metric failure. The comparison is only written when config, test-split, prompt-set, parser, and decoding digests all match.

Metric definitions: valid_json_rate — output parses into exactly one {"name": …, "arguments": …} call; argument_f1 — micro-F1 over arguments flattened to dotted key paths with canonical scalar JSON values (lists compared by index); exact-match metrics use canonical-JSON equality. Numerators and denominators for every metric, plus raw per-example records, are in `reports/`.

Prompt format

The adapter was trained and evaluated with one fixed prompt policy. Reproduce it exactly for best results:

• —system: the instruction below, then two newlines, then Available tools: and the tool schemas serialized as canonical JSON (sorted keys, compact ,/: separators):

  You are a tool-calling model. Select the correct tool and return only
  the JSON tool call. Do not include explanations.

  Available tools:
  [{"description":"…","name":"…","parameters":{…}}]

• —user: the raw request text.
• —The model answers with a one-element canonical JSON array: [{"arguments":{…},"name":"…"}].

Usage

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1"
ADAPTER = "abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM = (
    "You are a tool-calling model. Select the correct tool and return only "
    "the JSON tool call. Do not include explanations."
)
tools = [
    {
        "name": "lookup_weather",
        "description": "Look up the current weather for a city.",
        # xlam-style flat parameter map — the schema shape this adapter was
        # trained on. JSON-Schema-style {"type": "object", "properties": …}
        # tools are out of distribution and typically fail to parse.
        "parameters": {"city": {"description": "Name of the city.", "type": "str"}},
    }
]
tools_json = json.dumps(tools, separators=(",", ":"), sort_keys=True)
messages = [
    {"role": "system", "content": f"{SYSTEM}\n\nAvailable tools:\n{tools_json}"},
    {"role": "user", "content": "What is the weather in Paris today?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, do_sample=False, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# [{"arguments":{"city":"Paris"},"name":"lookup_weather"}]

Note add_special_tokens=False: the rendered chat template already contains the BOS token.

Training details

QLoRA with completion-only loss: every prompt token is masked with the ignore index and loss is computed only on the assistant tool-call tokens, with the prompt/target token boundary proven per example (training fails rather than falling back to full-sequence loss).

Full hyperparameters and the resolved run configuration are in `reports/config.resolved.yaml`; per-step metrics in `reports/training_metrics.jsonl`.

Data

Trained on the train split (15,000 examples) of abdelstark/sommelier-xlam-single-call-splits, a filtered, deduplicated, deterministically split derivative of Salesforce/xlam-function-calling-60k (CC-BY-4.0) containing only single-call examples. The validation split (1,000) was used for eval loss only; the test split (1,000) was never used for gradient updates. A normalized query appears in exactly one split.

Provenance and reproducibility

• —Pipeline: AbdelStark/sommelier, run nemotron-8b-full-3
• —Config digest: 3a8fb0d8b4f5b4c6bcde370a51b0111598814e28df78ba8e6c2c25a907e5958b
• —Test split digest: db8dd82f…; prompt set digest: a0da8fa2…
• —Machine-readable evidence: `reports/comparison_report.json`, per-model evaluation reports, runtime metadata (stage timings, peak GPU memory, cost marked unavailable rather than implied zero)

Intended use and limitations

Intended for research and engineering evaluation of schema-valid single-tool-call generation, and as a reference result for the sommelier pipeline.

• —The adapter emits one tool call per request. Multi-call plans, multi-turn tool use, and tool-result processing are out of scope; rows requiring several calls were excluded from training and evaluation.
• —Scoring is exact-match oriented: semantically equivalent but differently formatted argument values count as mismatches, so absolute argument metrics are conservative.
• —Tool schemas must use the xlam-style flat parameter map ("parameters": {"<param>": {"description": …, "type": …}}); JSON-Schema-style {"type": "object", "properties": …} schemas are out of distribution and typically produce unparseable calls.
• —Results hold for the recorded dataset revision, prompt policy, parser, and greedy decoding; do not read them as claims of production readiness, general agent reliability, or performance on other schemas.
• —English-only training data; tool schemas are synthetic (APIGen).
• —The model inherits the capabilities and biases of its base model; no additional safety tuning was performed.

License and attribution

Built with Llama.

• —Adapter weights are a derivative of nvidia/Llama-3.1-Nemotron-Nano-8B-v1 and are subject to the NVIDIA Open Model License and the Llama 3.1 Community License (the model name begins with "Llama" per that license).
• —Training data derives from Salesforce/xlam-function-calling-60k (CC-BY-4.0); see the dataset card for attribution and processing.
• —The sommelier pipeline code is MIT-licensed.

Citation

If you use this adapter, please cite the source dataset:

@article{liu2024apigen,
  title={APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author={Liu, Zuxin and Hoang, Thai and Zhang, Jianguo and Zhu, Ming and Lan, Tian and Kokane, Shirley and Tan, Juntao and Yao, Weiran and Liu, Zhiwei and Feng, Yihao and others},
  journal={arXiv preprint arXiv:2406.18518},
  year={2024}
}