79Labs/astraforge-8b-TCR

astraforge-8b-TCR — Tool-Calling & Retrieval Agent (LoRA on Llama-3.1-8B)

Developed by [79Labs](https://huggingface.co/79Labs) · Version 1.0.0

astraforge-8b-TCR is a LoRA adapter for `meta-llama/Llama-3.1-8B-Instruct`, trained for the behaviours that make a tool-using assistant trustworthy: choosing the right tool from a large catalog, asking for missing parameters, confirming before acting, emitting schema-valid calls, and answering only from the documents it was given.

• —Base: Llama-3.1-8B-Instruct
• —Adapter: LoRA r=16 on all attention + MLP projections, 160 MB
• —Training: 1 epoch over 100k rows of a synthetic agentic corpus (tool calling, elicitation, confirm-before-call, RAG, ReAct, guardrails); best eval loss 0.1530; 23h27m on a single GB10
• —Sibling: `79Labs/astraforge-70b-TCR`

The headline, and its limits. This adapter takes tool-calling correctness from 0.59 → 0.90 and confirm-before-acting from 0.00 → 0.89 against its own base, and on tool_correct it beats our 70B fine-tune (0.81) at a ninth of the size. It does not improve reasoning — GSM8K is inherited from the base and stays there. It also has a known prose defect, documented below, which we would rather you read than discover. Every number here has its raw evidence in `benchmarks/`.

What it is good at

In-house agentic benchmark (N=100)

Each model is prompted in its own native tool format. tool_correct = names the right tool with a schema-valid call within two turns; confirmed_first = asks for confirmation before calling.

Read it per metric against the base. confirmed_first is the one to look at hardest: a same-generation 4B fine-tune trained on comparable data scored 0.00 there, so 0.89 is evidence that the discipline was learned, not inherited. GSM8K −0.01 is inside the noise at N=100 (SE ≈ 0.04) — reasoning is a base property and this data cannot add it.

Evidence: `benchmarks/nway_results.json`.

RAG grounding (N=60, multi-document)

Each question is asked twice: once with the answer-bearing document present, once with it replaced by a distractor. The second condition is what separates grounding from memorisation.

Evidence: `benchmarks/rag_grounding.json`.

In an application

Driving a 105-tool assistant (retrieval → elicitation → confirmation → call), across 100 conversations with the tool, phrasing, answer order and interruptions all sampled at random: 90/100 completed with every invariant satisfied — never naming a tool or a parameter key (100/100), never executing before an explicit go-ahead (100/100), cancelling cleanly (12/12), applying a correction made at the summary (9/9), and returning to a parked task after a digression or a side request (16/16). Median turn latency 1.8 s in 4-bit on one GB10.

Evidence: `benchmarks/app_eval.json`, `benchmarks/conversation_acceptance.md`.

What it is not good at — please read this

1. It narrates in a machine voice on multi-tool requests

Asked to do two things at once, its user-visible prose opens with Step 1: open a new bank account in 40.8% of turns over a clean 105-tool catalog (15.2% / 6.1% over a broader one). The frame came from the training corpus, which still carries it on 15.5% of assistant turns; the generator was fixed afterwards and the corpus was never regenerated.

If you put this model in front of users, do not let it author the user-facing sentence for a tool flow. In our own application that text is rendered from the tool schema by code, and the leak rate across 100 sampled conversations is 0%. A corrected-narration revision is in progress and will ship as a separate version only if it passes a benchmark gate on toolcorrect and confirmedfirst.

Evidence: `benchmarks/machine_voice_appcatalog.txt`.

2. It does not generalise to foreign call syntaxes

On BFCL v4 `simple_python` (Prompt mode, N=400) it scores 37.50%, against 51.00% for the 70B on the identical cases. The breakdown matters: 198 of the 250 misses are `ast_decoder:decoder_failed` — the model emits the argument list without the function name ([number=5] where the 70B wrote [math.factorial(number=5)]). Only 3 are a wrong function name and 2 a missing required argument. So it is choosing correctly and formatting wrongly, in a Python-call syntax neither model was trained on. Prompt it in its native JSON format (as the in-house harness does) and it out-calls the 70B.

Evidence: `benchmarks/bfcl_v4_simple_python_score.json`.

3. Reasoning is the base model's

GSM8K 0.75. If you need stronger reasoning, this adapter will not give it to you.

4. Not measured

τ-bench, API-Bank, BFCL's other eleven categories and BFCL function-calling mode. Multi-language. Adversarial or hostile users. The sampled-conversation evaluation uses a simulated user that answers correctly and politely; typos, contradictions and multi-intent turns are outside it.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(model, "79Labs/astraforge-8b-TCR")
tok = AutoTokenizer.from_pretrained(base)

messages = [{"role": "user", "content": "Book a flight for Ada Lovelace from SFO to JFK on 2026-09-14."}]
ids = tok.apply_chat_template(messages, tools=[...], add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids.to(model.device), max_new_tokens=256)[0][ids.shape[1]:]))

Pass tools through the tokenizer's own chat template. The model is trained to answer with a {"name": ..., "arguments": {...}} object — and, when required parameters are missing, to ask for them first rather than invent them. That second behaviour is the point of the model; a harness that treats a clarifying question as a failed call will under-report it badly.

Intended use and limits

Built for business tool-calling assistants where a wrong call is expensive: ordering, booking, ticketing, account changes, lookups. Keep the guardrails in code — validate every call against its schema, and never execute without explicit user confirmation. The model is trained to support that discipline, not to replace it.

Not for: medical, legal or financial advice; autonomous execution without a human in the loop; any setting where the machine-voice defect above would reach a user unmediated.

License

Llama 3.1 Community License, inherited from the base model. Training data is synthetic and generated in-house.

Citation

@misc{astraforge8b2026,
  title  = {astraforge-8b-TCR: a small tool-calling and retrieval agent},
  author = {79Labs},
  year   = {2026},
  note   = {LoRA on Llama-3.1-8B-Instruct. Benchmarks, raw evidence and known defects included.},
  url    = {https://huggingface.co/79Labs/astraforge-8b-TCR}
}