Research map / Multi-agent systems and games / LLM agents
VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VerilogCoder, a multi-agent system that plans Verilog designs via a Task and Circuit Relation Graph and debugs functional errors with an AST-based waveform tracing tool, reaching 94.2% on VerilogEval-Human v2.
Problem
Generating functionally correct Verilog code from specifications with LLMs is hard because plans lose signal and state-transition details and existing LLM approaches cannot fix functional errors.
Approach
A multi-agent ReAct framework where an LLM planner builds a Task and Circuit Relation Graph (TCRG) of sub-tasks, signals, transitions and examples, retrieved by agents, and a debug agent back-traces mismatched output signals through the Verilog AST (via Pyverilog) using simulator waveforms.
Key results
- VerilogCoder (GPT-4 Turbo) reaches 94.2% pass rate on VerilogEval-Human v2, versus 60.3% for GPT-4 Turbo baseline.
- VerilogCoder (Llama3-70B) reaches 67.3% pass rate, exceeding Llama 3 (41.7%).
- Ablation: AST-WT adds 11.5% with Planner1 and 27.5% over baseline with Planner2 (TCRG planner + AST-WT); TCRG planner alone adds 7.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VerilogCoder with GPT-4 Turbo instead of GPT-4 Turbo prompting | GPT-4 Turbo · VerilogEval-Human v2 | pass-rate: 60.3 -> 94.2 | Helped |
| use VerilogCoder with Llama3 instead of Llama 3 prompting | Llama 3 · VerilogEval-Human v2 | pass-rate: 25.6% | Helped |
| add AST-WT to Planner1 (ablation) | Planner1 · VerilogEval-Human v2 | pass-rate: 66.7% -> 78.2%; 11.5% | Helped |
| replace Planner1 with Planner2, without AST-WT (ablation) | VerilogCoder · VerilogEval-Human v2 | pass-rate: 66.7% -> 74.4%; 7.7% | Helped |
| combine Planner2 with AST-WT (ablation) | VerilogCoder · VerilogEval-Human v2 | pass-rate: 66.7% -> 94.2%; 27.5% | Helped |
| use Planner1 with AST-WT instead of Planner2 without AST-WT (ablation) | VerilogCoder · Application (Descr.) | pass-rate: 10.5% | Helped |
| use Planner1 with AST-WT instead of Planner2 without AST-WT (ablation) | VerilogCoder · Comb+Seq+FSM (Descr.) | pass-rate: 39.1% | Helped |
| use Planner1 with AST-WT instead of Planner2 without AST-WT (ablation) | VerilogCoder · Comb+Seq+FSM (Waveform) | pass-rate: 12.5% | Helped |
| use Planner2 without AST-WT instead of Planner1 with AST-WT (ablation) | VerilogCoder · Comb (Kmap) | pass-rate: 33.3% | Helped |
| use Planner2 without AST-WT instead of Planner1 with AST-WT (ablation) | VerilogCoder · FSM (Trans. Table) | pass-rate: 44.5% | Helped |
Surprising findings
- Planner2 without AST-WT outperforms Planner1 with AST-WT on combinational Kmap and FSM transition-table tasks by 33.3% and 44.5%.
- Average token count of VerilogCoder is roughly 13x the GPT-4 Turbo baseline.
Limitations
- Each problem run only once, with a single benchmark (156 problems) evaluated.
- High token cost (about 13x GPT-4 Turbo baseline).
- Baseline numbers for prior LLMs are taken as best over several prompting settings, making comparisons less controlled.
Experimental setup
Datasets: VerilogEval-Human v2
Benchmarks: VerilogEval-Human v2
Metrics: pass rate (pass@1)
Baselines: RTL-Coder DeepSeek Coder CodeGemma CodeLlama Llama 3 Mistral Large GPT-4 GPT-4 Turbo
Builds on: ReAct AutoGen Pyverilog iverilog
Closest papers by meaning
- BetterV: Controlled Verilog Generation with Discriminative Guidance (ICML 2024 · similarity 0.89)
- VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification (NeurIPS 2025 · similarity 0.89)
- QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation (NeurIPS 2025 · similarity 0.87)
- DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model (ICLR 2025 · similarity 0.86)
- CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair (ICLR 2025 · similarity 0.85)
- Circuit Representation Learning with Masked Gate Modeling and Verilog-AIG Alignment (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.