Research map / Large language models / Agents, code and math

CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

ICLR 2025 · Large language models › Agents, code and math · approach: Code generation benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds correct-by-construction Verilog data for Karnaugh maps, FSMs and waveforms plus error-report-driven targeted code repair data, and fine-tunes Starcoder2-15B to reach SOTA on VerilogEval and RTLLM.

Problem

LLM-generated synthetic Verilog data is noisy: models struggle with non-textual representations (Karnaugh maps, state diagrams, waveforms) and show unstable pass rates across training checkpoints due to minor errors that cannot be verified without testbenches.

Approach

Generate non-textual Verilog problems and solutions correct-by-construction from templates (with simulated VCD waveforms as testbench outputs), and create targeted repair data by having an LLM write error reports from failing completions, validating them, and injecting similar errors into open-source code.

Key results

• Starcoder2-15B fine-tuned on SDG-CC-Repair reaches 81.9% pass@1 on VerilogEval-Machine and 68.0% on VerilogEval-Human, versus prior SOTA gains of 3.8 and 10.9 points.
• On RTLLM v1.1, the SDG-CC-Repair Starcoder2-15B model achieves 65.8% functional pass@5, a 6.6-point improvement over prior SOTA as reported by the authors.
• Ablation: removing self-consistency validation drops VerilogEval-Human pass@1 from 68.0 to 63.3, and removing the error report drops it to 59.6.

Measured effects reported in the paper

Surprising findings

• Roughly 15% of VerilogEval-Human problems showed large pass-rate discrepancies between consecutive checkpoints, even though training loss decreased steadily.
• Directly prompting the LLM without the error report resolved only 13% of errors, versus much higher success with the report-guided hints.

Limitations

• Generated code diversity is low for non-textual problems, reflecting template-based CC data.
• Model performance remains weaker on waveform problems, which require reverse-engineering circuits from waveforms.
• RTLLM results use Icarus Verilog, which may not support all syntax, so numbers are not directly comparable to VCS-based prior work.

Experimental setup

Datasets: The Stack v2 (Verilog/SystemVerilog subset) Wikipedia textbooks VerilogEval-Human non-textual problems as templates

Benchmarks: VerilogEval-Machine VerilogEval-Human RTLLM v1.1 VerilogEval-NonText

Metrics: pass@1 pass@5 pass@10 syntax pass rate functional pass rate BLEU Jaccard TSED

Baselines: CodeV OriGen BetterV RTLCoder GPT-4o Llama-3.1 DeepSeek-Coder-V2 Starcoder2

Builds on: Self-Instruct OSS-Instruct VerilogEval RTLCoder CodeV

Closest papers by meaning

• VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification (NeurIPS 2025 · similarity 0.88)
• DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model (ICLR 2025 · similarity 0.87)
• QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation (NeurIPS 2025 · similarity 0.87)
• BetterV: Controlled Verilog Generation with Discriminative Guidance (ICML 2024 · similarity 0.86)
• LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.86)
• Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.86)