Modelpublic

immortaltatsu/ghostai-lfm2.5-1.2b-app-v2

sourceHugging Faceotherupdated 27d agoView on Hugging Face

0likes511downloads

Model Card

GhostAI LFM2.5-1.2B — app-contract (eval-2)

On-device tool-calling model for the GhostWallet Solana app. Given the app's system prompt and a retrieved slice of its tool catalog, it emits a single Hermes-style tool call, then answers in one line from the returned tool result.

<tool_call>{"name":"get_wallet_balance","arguments":{}}</tool_call>

This drop is trained off the 1000-question iOS UI eval (agentic-eval-2) plus the full 174-tool teacher corpus. Epoch 1 only — later epochs overfit (eval loss 0.153 → 0.171 → 0.187).

Versus the previous weights on this repo (v3): argument validation 85.7% → 91.1%, prompt-injection 88.9% → 100%, tool-correct 35.7% → 37.5%. Confirm-gate bypasses stay at 0. Overall e2e is 67.1% (same as v3; 3 cases gained, 3 lost).

Files

File	Size	Notes
`model.safetensors`	2.3 GB	bf16, epoch 1 (step 197)
`ghostai-lfm-app-v4-F16.gguf`	2.3 GB	full-precision reference
`ghostai-lfm-app-v4-Q4_K_M.gguf`	698 MB	what runs on device (llama.rn / llama.cpp)

ghostai-lfm-app-v3-*.gguf on this repo are the same epoch-1 weights, kept so existing download URLs keep working.

Results — app harness (Q4KM, greedy, llama.cpp)

Single-turn tool selection, 56 held-out utterances:

metric	v2	v3	this
emitted a parseable tool call	91.1%	92.9%	92.9%
arguments passed app validation	71.4%	85.7%	91.1%
correct tool	23.2%	35.7%	37.5%
correct tool, when retrieval offered it	50.0%	83.3%	83.3%

End-to-end, 76 cases / 88 turns (real ChatSession.send() loop):

metric	v2	v3	this
overall	60.5%	67.1%	67.1%
grounding	50.0%	58.8%	55.9%
value-moving gate respected	75.0%	91.7%	91.7%
prompt-injection resistance	94.4%	88.9%	100%
multi-turn	25.0%	33.3%	25.0%
confirm-gate bypasses	0	0	0
planted content reaching a tool argument	0	0	0
turns with an invented number	9	1	1

Hash-embedding retrieval still only offers the gold tool ~10.7% of the time on this single-turn set; tool_correct is capped by that.

What changed

—174/174 app tools in the corpus (v3 had 69). Teacher traces merged with hand-authored discrimination pairs.
—Eval-2 UI failures: slang holdings (yo how much sol i got, ghost show me my bag), pay X.sol / trade X for Y, agent spawn/sweep/policy, typo register.
—AGENTS.md refusals: no seed / mnemonic / private-key echo; value-moving answers wait for slide-to-confirm and never claim a broadcast.
—Sibling-tool distractors boosted into the same retrieval window so the model learns the choice (send_sol vs send_token, execute_swap vs get_swap_quote).

Contract

Unchanged: the model expects the app's assembled system prompt, emits exactly one <tool_call> block and no prose, then answers in one short line once the app appends the tool result. It is not a planner — the host app owns tool retrieval, argument validation, and the slide-to-confirm gate for anything that moves value.

Training

—Base: LFM2.5-1.2B-Thinking
—7,631 traces, 174 tools, split by utterance template
—3 epochs, assistant-only loss, 2× A100 ZeRO-3 — weights here are epoch 1
—Eval loss: 0.153 / 0.171 / 0.187 (epoch 1 / 2 / 3)

Limitations

1.The 1000-case UI eval is not a clean holdout. 60 of those prompts appear verbatim in train (wallet slang, jailbreaks, typos). Score the other 935 for an honest number.
2.Grounding is 55.9%, slightly behind v3's 58.8%.
3.Multi-turn is 25%, same as v2, below v3's 33%.
4.Retrieval is still the bottleneck (~11% primary in top-5 on hash fallback).
5.Synthetic data only. No real user transcripts.
6.Do not run without the host confirm gate. Gate bypasses were 0 on this harness; that is not a safety guarantee.

License

Derived from LFM2.5-1.2B-Thinking and distributed under the LFM Open License v1.0 — see LICENSE. Verify commercial terms with Liquid AI before production use.