Modelpublic

emailvenky/chia-governor-triage-7b

sourceHugging Facebsd-3-clauseupdated 4d agoView on Hugging Face

0likes15downloads

Model Card

chia-governor-triage-7b

A LoRA adapter for Qwen2.5-Coder-7B-Instruct that predicts, from the code alone, whether a Verilog design change will fail its test bench before any simulator is run.

It exists to sit in front of the expensive evaluation stage of an agentic hardware design loop. Simulation is slow and costly, and many agent proposals are simply wrong. This model triages them.

Why it beats larger models at this task

Every frontier model tested carries the same wrong prior: that any edit to working code is probably a bug. That produces near-perfect recall and poor precision, so the gate throws away a large share of perfectly good designs.

This adapter was trained on data containing verified-harmless edits alongside real faults, which attacks that bias directly.

Accuracy by edit family	Untrained	Tuned
benign (behaviour preserved)	55.6%	94.8%
faulty (behaviour changed)	65.6%	79.4%

Near chance to near ninety-five on harmless edits. That 39-point gain is the whole result.

Results

630 held-out examples, from VerilogEval problems absent from training entirely. Positive class is FAIL.

Model	Accuracy	Precision (FAIL)	Recall (FAIL)	Cost / 1k	Latency
Qwen2.5-Coder-7B, untrained	61.4%	56.1%	73.4%	$0.049	0.25s
This adapter	85.2%	83.3%	84.8%	$0.063	0.32s
Gemini 2.5 Flash	86.2%	77.9%	97.6%	$0.163	6.47s
Gemini 2.5 Pro	81.5%	71.5%	99.7%	$0.676	14.4s
Gemini 2.5 Flash-Lite	68.1%	59.3%	96.9%	$0.054	0.42s

Majority-class baseline is 54.1%.

Precision is the measure that matters here. A false FAIL silently discards a working design, which hurts a design loop more than paying for one extra simulation. This adapter has the highest precision of any model tested, at roughly a quarter of Flash's cost and a twentieth of its latency. Flash retains clearly better recall.

Tunable operating point

Threshold	Accuracy	Precision	Recall	Sims skipped	Good designs lost
0.50	85.2%	83.3%	84.8%	294	49 of 341
0.70	83.7%	86.9%	75.8%	252	33 of 341
0.90	80.2%	92.7%	61.6%	192	14 of 341
0.95	77.0%	95.0%	52.6%	160	8 of 341

Usage

python

from chia_governor import TriageGate, LocalScorer

gate = TriageGate(
    LocalScorer("Qwen/Qwen2.5-Coder-7B-Instruct",
                adapter="emailvenky/chia-governor-triage-7b",
                load_4bit=True),
    threshold=0.9,
)

if gate.allow(spec, candidate_code):
    result = run_simulation(candidate_code)

Or directly. Score by comparing the PASS and FAIL logits rather than generating text, which is deterministic and cannot return an unparseable answer:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "emailvenky/chia-governor-triage-7b")

The prompt format is fixed and must match training: a system message describing the task, then the specification and the candidate code, asking for one word. See chia_governor/triage_gate.py for the exact strings.

Training


Base	`Qwen/Qwen2.5-Coder-7B-Instruct`
Method	LoRA, rank 16, alpha 32, dropout 0.05
Trainable	40.4M parameters (0.53%)
Quantisation	4-bit NF4
Data	1,941 examples, labels from real Icarus Verilog runs
Epochs	2 (244 optimizer steps)
Hardware	One NVIDIA L4, 72 minutes

Loss fell from 0.23 and plateaued near 0.10 rather than collapsing toward zero, which is the shape of a learned decision rule rather than memorisation.

Data

Generated rather than collected, with labels correct by construction. Working VerilogEval references are edited by script and every variant is simulated against its real test bench. Two families: eight faulty operators (flipped comparisons, swapped operators, off-by-one constants, altered bit ranges, changed reset values, non-blocking to blocking, dropped assignments, swapped signals) and six benign ones (commutative swaps, internal renames, redundant parentheses, reformatted literals, reordered independent assigns, dead wires).

Both families are necessary. A first pass with faulty edits only produced a set that was 79.7% FAIL, scoreable at 80% by always answering FAIL. Adding benign edits brought it to 44.8% and made the task meaningful.

Train and test are split by problem, so no VerilogEval problem appears on both sides.

Limitations

—Recall is the weak dimension. At 84.8%, roughly one failing candidate in six gets through, where Gemini Flash catches it.
—Faults are synthetic. They imitate plausible agent errors but are not drawn from real agent output.
—Small modules only. VerilogEval modules are far smaller than an industrial RTL block; transfer is untested.
—Contamination is plausible. VerilogEval may appear in pretraining data. The mutated variants do not, but the base problems may.
—One run, no seeds. No variance reported.

Licence

BSD-3-Clause, matching CHIA.