LoganLeGrand/Phi-4-mini-instruct-ShopRAG

Phi-4-mini-instruct-ShopRAG

A post-trained Phi-4-mini-instruct (3.8B) for edge-device RAG against CNC machine tool documentation and live backup data. Built for FANUC 31i-Model B controllers. Two-stage training: SFT on ~600 grounded examples, then DPO on 102 contrast pairs targeting residual safety failures.

The model answers from retrieved documentation and decoded machine state only. It does not answer from memory. When backup data is absent, it says so instead of guessing.

Model Summary

Evaluation

148 held-out shop-floor questions, binary scoring on factual correctness and grounding.

The frontier models fail because they know too much — they inject real-sounding FANUC facts that aren't in the retrieved context and may not apply to the specific controller model or builder customization. The fine-tuned model has no FANUC knowledge to leak; it reads what's in front of it.

Deployed as GGUF via llama.cpp / LM Studio. Two quantizations shipped:

Naive q4km (no imatrix) corrupted program lines — the domain-calibrated importance matrix fixed it by telling the quantizer which weights matter most for exact string relay.

Corpus

17,649 structured records extracted from six FANUC manual categories using NuExtract3 (NuMind, 4B params, Apache 2.0, Qwen3.5-4B base). NuExtract3 is a template-driven extraction model — you give it a JSON schema and source text, it fills in the fields verbatim. It extracts rather than paraphrases, which matters for technical manuals.

NuExtract3 runs locally via LM Studio (or any OpenAI-compatible server). Each call sends a system prompt with extraction rules and a user prompt containing the source text + a JSON template with empty fields. The model returns the template filled in. For parameter manuals the template includes number, name, data_type, bits (with when_0/when_1 per bit position), and text. For section-shaped manuals (operator, maintenance, PMC) it includes title, section_type, text, steps, and warnings.

PDFs are segmented into blocks (one per parameter or section heading), each block is extracted via NuExtract3 with category-specific prompts, validated for domain constraints (bit range, spurious boundaries, cross-contamination), then embedded into a LanceDB hybrid search index.

Record schema

Two shapes — parameter records carry bit-level definitions with 0/1 states, section records carry procedures/warnings/troubleshooting steps. Both use the same flat structure:

{
  "id": "<controller>::param::<number>::<manual_id>::pg<page>",
  "type": "parameter",
  "controller": "<controller_model>",
  "category": "NC_Parameters",
  "number": "9999",
  "name": "Example configuration bits",
  "data_type": "Bit",
  "text": "Bit 4 (EXMx): Example mode select ...",
  "bits": [
    {"bit": 4, "name": "EXMx", "when_0": "Mode A is selected", "when_1": "Mode B is selected"}
  ],
  "source": {"file": "<manual_pdf>", "page": [100, 101]}
}

Backup decoder

CNC backup files (e.g. CNC-PARA.TXT) store parameter values as raw strings. For bit-type parameters, the value is an 8-digit binary string where each digit maps to a bit position (bit 7 is leftmost, bit 0 is rightmost):

Raw backup value for parameter 9999:  00010000
                                      |||||||└─ bit 0
                                      ||||||└── bit 1
                                      |||||└─── bit 2
                                      ||||└──── bit 3
                                      |||└───── bit 4 = 1  ← EXMx
                                      ||└────── bit 5
                                      |└─────── bit 6
                                      └──────── bit 7

The decoder reads the bit value, matches it against the record's when_0/when_1 definitions, and produces a > **Current:** callout:

> **Current:** Bit 4 (EXMx) = 1 — Mode B is selected.

For non-bit parameters (word, real, etc.), the raw value is decoded directly — units are resolved against the machine's actual unit setting (mm vs inch) when the record documents both.

This decode step is done by software at query time, not by the model. The decoded callout is injected into the rendered context block so the model only needs to relay it.

At retrieval time, records are rendered into a fixed markdown shape with definition tables, > **Current:** callouts for live backup values, and > **Changed:** callouts for diff history. The model sees this same shape in every training example.

Note: The training dataset and extracted corpus are not included in this release — they contain proprietary FANUC documentation. To reproduce this approach for your own controllers, you would need to build your own extraction pipeline against your own manuals and backups.

Training Data

SFT — 523 train + 8 seed, 71 held-out eval

Every training example was generated through the real retrieval pipeline against real manuals and real backup data, then audited by a grounding verifier that rejected ~10–15% of attempts for fabrication. Questions are written the way machinists talk — terse, conversational, sometimes misspelled.

Refusal categories make up ~18% of training — deliberately overweighted. An early pilot showed safety behaviors need dense repetition to stick.

DPO — 102 pairs

Targeted the failure modes SFT left behind. The main one: reading a documentation table and presenting it as live machine state when no backup was in context. After DPO, that failure was gone across all test runs.

Prompt Template

Standard Phi-4 chat format. The **Active:** line tells the model what sources are present so it never has to guess.

<|system|>You are the CNC documentation and machine-state assistant for this shop.

Answer using only the documentation and machine data provided below — never invent parameter behavior, bit meanings, or machine settings that aren't shown to you.

If the information needed to answer isn't in what's provided, say so plainly, and name which source category the user could enable for a fuller answer.<|end|><|user|>**Active:** Parameter manual [✓/✗] · CNC-PARA backup [✓/✗]

[Rendered context blocks]

## Question
[User question]<|end|><|assistant|>

Example: backup active

<|user|>**Active:** Parameter manual ✓ · CNC-PARA backup ✓

### Parameter 9999 — Example configuration bits
**Layer:** Controller · **Type:** Bit path

| Bit | Name | Value 0 | Value 1 |
|---|---|---|---|
| 4 | EXMx | Mode A is selected | Mode B is selected |

> **Current:** Bit 4 (EXMx) = 1 — Mode B is selected.

*Source: Example Parameter Manual, p.100*

## Question
What's parameter 9999 bit 4 set to on this machine?<|end|><|assistant|>

Response:

Parameter 9999 bit 4 (EXMx) is set to **1** on this machine — Mode B is selected. When set to 0, Mode A would be selected instead.

Example: backup inactive (refusal)

<|user|>**Active:** Parameter manual ✓ · CNC-PARA backup ✗

### Parameter 9999 — Example configuration bits
[same documentation table, no Current: callouts]

## Question
Is Mode B active on this machine right now?<|end|><|assistant|>

Response:

Parameter 9999 bit 4 (EXMx) controls whether Mode B is active (1) or Mode A (0). I can't tell you the current setting — the CNC-PARA backup isn't active. Enable it to check the live value.

Inference

Sampler settings (critical)

Pin these exactly. Default repeat_penalty 1.1 in LM Studio / llama.cpp corrupts exact table relays and program-line reproduction.

llama-server

llama-server.exe -m phi4mini-shoprag-4bb-q4km-imx.gguf --port 8765 -ngl 99 --parallel 1 -c 4096 --no-webui

Python (local server API)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8765/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="phi4mini-shoprag-4bb",
    messages=[
        {"role": "system", "content": "You are the CNC documentation and machine-state assistant for this shop.\n\nAnswer using only the documentation and machine data provided below — never invent parameter behavior, bit meanings, or machine settings that aren't shown to you.\n\nIf the information needed to answer isn't in what's provided, say so plainly, and name which source category the user could enable for a fuller answer."},
        {"role": "user", "content": "**Active:** Parameter manual ✓ · CNC-PARA backup ✗\n\n### Parameter 5006 — Tool length compensation bits\n..."}
    ],
    temperature=0.0,
    extra_body={"repeat_penalty": 1.0, "top_k": 0, "min_p": 0},
    max_tokens=800
)

print(response.choices[0].message.content)

Hardware

~3.3 GB total. Runs on a single GPU with 8+ GB VRAM. CPU-only works as a slower fallback. No cloud, no internet, no subscription. Response time ~2–3 seconds on a mid-range GPU.