CoolFace
Modelpublic

turnercore/needle2-automaticity-v9

sourceHugging Faceotherupdated 1mo agoView on Hugging Face
0likes
Model Card

Needle 2 — Automaticity V9 (negative result)

This is an experimental Cactus Needle 2 archive fine-tuned for Automaticity V9 function calling. Do not deploy it as a replacement for Needle Point / `ln-point-v9`. It is published for reproducibility of a completed negative result.

The artifact is a Cactus-engine .cact archive, not a Transformers or PyTorch checkpoint. It requires the pinned official Needle 2 engine family. It deliberately does not include the pickle LoRA adapter, training rows, benchmark rows, predictions, or environment files; those are retained in the private evidence repo.

Identity

ItemValue
Upstream Needle sourcecactus-compute/needle@571fcd68f48b6649c91f353eb91d0a0c9b9135ce (v2.0.9)
Official engine2.0.3
Base modelCactus-Compute/needle2@98fbd955b0347e78059be0c253cc1ffa09b87bc7
Training4,900 V9-native rows; 1 epoch; rank-16/alpha-32 LoRA; LR 1e-4; seed 0
Export405 tensors; 13,737,807 bytes
needle2-automaticity-v9.cact SHA-2564f19e7fb27e8c11780df8f2497b26dd96aef68b112c1a158a5e2fec68f4d7c70
Canonical implementationturnercore/ai-gateway@1313c64c7a04232af7f1b5534a902d7f63ffaf50

Sealed Automaticity V9 benchmark v1.1

The unchanged private benchmark had 1,050 rows and SHA-256 c24db4269886018a400996d4d5e7c6f5ad9fad31ead5c6aab2a90217f691c9bb. Needle 2 and the retained Needle Point reference used the same rows, scorer fields, and semantic tool catalogues. Benchmark rows and labels were never used for training, prompt selection, decoding selection, or conversion tuning.

MetricNeedle Point v1Needle 2 FT
End-to-end exact96.10%70.38%
Routing / contract match98.95%70.76%
Capability accuracy100%100%
Argument accuracy96.19%74.95%
Action exact86.86%0.32%
Schema-valid output100%100%
No-tool precision / recall99.46% / 100%70.69% / 100%
Wrong-tool / wrong-argument11 / 30307 / 4

Needle 2 preserved structural validity but mostly returned no tool where an action was required. The candidate retrieval stage retained the gold tool for all action rows, so this is not a retrieval-recall issue.

Runtime caveat

Needle Point used a warm persistent PyTorch ROCm runtime (324.81 ms average); Needle 2 used the official CPU engine with per-row agent binding (875.30 ms average). These are runtime-specific operational measurements, not a pure model-speed comparison. The tuned Needle 2 archive reports confidence: null by upstream design because the confidence head is not fine-tuned.

Files

  • needle2-automaticity-v9.cact: final runnable archive.
  • ab-comparison.json: aggregate metrics, failure counts, identities, and runtime confounders; SHA-256 c9ae921d5310722df0d48528715f41115c4b4a3463a628e98d2c443f4acc8905.
  • provenance.json: safe, machine-readable artifact and source identities.
  • artifact-runtime-smoke.json: sanitized representative official-engine smoke.

The native dataset, selection audits, benchmark rows, predictions, LoRA adapter, and detailed execution evidence live in the private dataset repo turnercore/automaticity-v9-needle2-experiment.