Research map / Multi-agent systems and games / LLM agents

Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Rule2DRC, a 1,000-task benchmark with 13,921 evaluation layouts for natural-language-to-DRC-script synthesis with execution-based scoring, and SplitTester, an execution-guided tester agent that splits high-scoring indistinguishable candidate clusters to improve Best-of-N selection.

Problem

Translating natural-language design rules into executable DRC scripts is labor-intensive, and existing benchmarks are small, score by code similarity rather than execution, or require labeled test layouts as agent input.

Approach

Generate discriminative test layouts by repeatedly targeting the largest high-scoring cluster of candidate scripts that produce identical outputs, reclustering after each new test, instead of pruning to a top-3 set early or only refining already-separated groups.

Key results

• SplitTester at BoN-20 on GPT-OSS-120B reaches 63.8% success versus 62.7% for CodeMonkey and 59.3% for S*, with error 4.1%.
• On GPT-OSS-20B at BoN-20, SplitTester improves success from 41.1% (CodeMonkey) to 44.4%, with error reduced from 14.1% to 12.6%.
• Including the 60K-token KLayout API documentation raises GPT-OSS-120B pass@1 by over 20 points and pass@20 by over 40 points.

Surprising findings

• LLM-as-a-Judge selection performs far below test-based selection, with success rates of 15.7% to 37.6% depending on model.
• Label error of generated tests is 15.7% to 37.6% across models, yet the expected labels still help selection.

Limitations

• Expected labels generated by the LLM are noisy, with label error between 15.71% and 37.60%.
• Only selection-time components of baselines were compared, not full S* or CodeMonkey pipelines.
• Synthetic rules were drafted by GPT-5.2 and manually reviewed, which may bias the benchmark.

Experimental setup

Datasets: SkyWater130 PDK rules KLayout API documentation

Benchmarks: Rule2DRC

Metrics: success rate error rate Oracle@N pass@N

Baselines: LLM-as-a-Judge Generated Tests S* CodeMonkey Sample-1

Builds on: CodeMonkey S* Best-of-N sampling CodeT

Closest papers by meaning

• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.84)
• From I/O to Code with Discovery Agent (NeurIPS 2026 · similarity 0.84)
• Property-Guided LLM Program Synthesis for Planning (NeurIPS 2026 · similarity 0.84)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.83)
• SymRTLO: Enhancing RTL Code Optimization with LLMs and Neuron-Inspired Symbolic Reasoning (NeurIPS 2025 · similarity 0.83)
• WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch (NeurIPS 2025 · similarity 0.83)