Research map / Reinforcement learning / Visual and representation RL
A Hierarchical Adaptive Multi-Task Reinforcement Learning Framework for Multiplier Circuit Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HAVE, a hierarchical adaptive multi-task RL framework (Pareto-driven meta-agent plus factored multi-task Q-network) that designs multipliers across the full preference space, beating RL-MUL and other baselines in hypervolume.
Problem
Multi-objective multiplier circuit design (area vs. delay) requires finding Pareto-optimal designs across all objective preferences, which existing RL methods do not do sample-efficiently.
Approach
A meta-agent (trained with policy gradient) generates diverse, delay-constraint-augmented preferences guided by the Pareto hypervolume reward, while one shared preference-conditioned multi-task Q-network with a factored (column/type) Q-decomposition and restore-then-explore exploration optimizes the multiplier designs for those preferences.
Key results
- HAVE improves hypervolume by up to 28.08% over RL-MUL on multiplier design problems.
- HAVE improves hypervolume by up to 89.05% over Wallace on 16-bit Booth multipliers (Table 1 ablation).
- On PE arrays built from HAVE multipliers, hypervolume improves by up to 28.84% over RL-MUL and maximum delay reduction of 10.26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add restore-then-explore to RL-MUL (ablation) | RL-MUL · 32-bit Booth | hypervolume: 2272.40 -> 2329.80 | Helped |
| add factored Q-value learning to R (ablation) | R · 16-bit Booth | hypervolume: 373.50 -> 397.32 | Helped |
| add factored Q-value learning to R (ablation) | R · 32-bit Booth | hypervolume: 2329.80 -> 2395.31 | Helped |
| add static multi-task learning to FR (ablation) | FR · 16-bit Booth | hypervolume: 397.32 -> 410.34 | Helped |
| add static multi-task learning to FR (ablation) | FR · 32-bit Booth | hypervolume: 2395.31 -> 2409.01 | Helped |
| add the hierarchical meta-agent to MFR (ablation) | MFR · 16-bit Booth | hypervolume: 410.34 -> 416.07 | Helped |
| add the hierarchical meta-agent to MFR (ablation) | MFR · 32-bit Booth | hypervolume: 2409.01 -> 2414.17 | Helped |
| use HAVE rather than Wallace | Multiplier compressor tree · 8-bit Booth | minimum delay: 0.9652 -> 0.9117 | Helped |
| use HAVE rather than Wallace | Multiplier compressor tree · 16-bit Booth | minimum delay: 1.4980 -> 1.3593 | Helped |
| use HAVE rather than Wallace | Multiplier compressor tree · 16-bit Booth | minimum area: 1867 -> 1723 | Helped |
Surprising findings
- Some compressor trees designed by HAVE have more stages and compressors yet lower physical delay and area, highlighting the gap between analytical and synthesis metrics.
Limitations
- Evaluated only on multiplier design (and PE arrays built from them); generality to other circuits is argued, not shown.
- Relies on slow synthesis-in-the-loop reward, making sample efficiency critical.
- Factored Q-function space is a restricted subspace; convergence guarantee depends on a projection error term.
Experimental setup
Benchmarks: 8/16/32/64-bit AND and Booth multiplier design problems PE array with multipliers
Metrics: hypervolume area delay
Baselines: Wallace Tree GOMIL RL-MUL RL-Scalar Random
Builds on: RL-MUL deep Q-network universal value function approximators asynchronous Q-learning Go-Explore
Closest papers by meaning
- Reinforcement Learning within Tree Search for Fast Macro Placement (ICML 2024 · similarity 0.86)
- Towards Automated RISC-V Microarchitecture Design with Reinforcement Learning (AAAI 2024 · similarity 0.86)
- AUTOCIRCUIT-RL: Reinforcement Learning-Driven LLM for Automated Circuit Topology Generation (ICML 2025 · similarity 0.86)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.85)
- Retrieval-Guided Reinforcement Learning for Boolean Circuit Minimization (ICLR 2024 · similarity 0.85)
- Reinforcement Learning Policy as Macro Regulator Rather than Macro Placer (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.