Modelpublic

Pablo-Flores-Mollinedo/verilog-llm-rigorous-rtl-benchmark-report

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes

Model Card

Verilog LLM rigorous RTL benchmark artifacts

Paper-oriented benchmark/report artifact repo for Verilog LoRA and verified-agentic RTL experiments.

Highlight results already verified

text

Best clean single adapter: v33 Qwen3.5 LoRA
VerilogEval v2 spec-to-RTL direct n=1: compile 113/156, pass 80/156

Previous practical workflow: v29 v9-primary specialist PEFT-MoE selector
VerilogEval v2 spec-to-RTL: compile 150/156, pass 84/156

Best practical workflow: v36 verified agentic pipeline
VerilogEval v2 spec-to-RTL max B=5 model calls/problem: compile 145/156, pass 95/156

Important: v29 and v36 are not standalone adapters and not zero-shot single-model pass@1. They are verified workflows using iverilog/vvp feedback/selection.

Running rigorous matrix

Active matrix includes:

text

v9_direct       Qwen2.5-Coder-7B + v9 baseline LoRA
v30b_direct     Qwen2.5-Coder-7B + v30b LoRA
v33_direct      Qwen3.5-9B + v33 LoRA
v34_direct      Qwen3.5-9B + v34 manual repair LoRA
v29_selector    Qwen2.5 v9-primary specialist PEFT-MoE verifier selector
v36_agentic     Qwen3.5 v33/v34 verified repair loop

Benchmarks:

text

VerilogEval v2 spec-to-RTL
VerilogEval v2 code-complete ICCAD 2023
paper-style 30-task functional suite       # direct adapters only
RTLLM v2 verified-reference subset         # direct adapters only

Default protocol:

text

3 repeats per system/benchmark
temperature=0, top_p=0.01
every RTL source recompiled and simulated

Current running status is in:

text

documentation/paper_rigorous_running_status.md

Final CSV/plots will be uploaded here when matrix completes.

Included files

text

documentation/paper_rigorous_evaluation_plan.md
documentation/paper_rigorous_running_status.md
scripts/run_paper_rigorous_matrix.sh
scripts/run_rtllm_direct_benchmark.py
scripts/aggregate_paper_rigorous_results.py
scripts/run_v29_runtime_primary_specialist_selector.py
scripts/run_v36_agentic_verified_pipeline.py
results/v36_agentic_verified_pipeline/*
documentation/sources/papers/*.pdf

Correct citation/claim style

Use separate tables for:

text

direct single-adapter pass@1: v9/v30b/v33/v34
verified selector workflow: v29
verified agentic workflow: v36
external RTLLM/paper-style generalization: direct adapters only