Research map / Label-efficient and robust learning / Recognition and long-tail
Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-level hierarchical Mixture of Experts (node, basic-block, and graph MoEs aggregated by a high-level MoE) on top of the HARP GNN, with a two-stage training scheme, to improve domain generalization for HLS pragma design prediction.
Problem
HLS performance prediction models for FPGA pragma design generalize poorly to new C++ kernels because of large domain shift and scarce HLS labels for new kernels.
Approach
Apply mixture-of-experts at three natural program granularities (node, basic block/pseudo node, graph) inside a GNN and aggregate the three low-level MoEs with a high-level gating network, trained with a two-stage schedule (expert warmup, then alternating joint/expert training) to avoid expert polarization.
Key results
- Hierarchical MoE reaches total offline MSE 0.143 vs 0.202 for HARP on target kernels.
- Hierarchical MoE achieves 1.38 geometric mean FPGA speedup vs AutoDSE, versus 1.09 for HARP (about 26.6% higher).
- On 5 complex kernels, hierarchical MoE gets MSE 0.401 vs 0.426 for HARP and 3.227 for HARP+MAML.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace HARP with hierarchical MoE | HLSyn target kernels · HLSyn target kernels | Total MSE: 0.202±0.013 -> 0.143±0.028 | Helped |
| replace HARP with hierarchical MoE | HLSyn target kernels · HLSyn target kernels | Geo mean speedup: 1.09 -> 1.38 | Helped |
| replace HARP with hierarchical MoE | complex kernels · Complex kernels | MSE: 0.426 -> 0.401 | Helped |
| combine node, block, and graph granularities with hierarchical aggregation rather than putting MoEs together in one model (ablation) | HLSyn target kernels · K-means fine-tuning split | MSE: 0.174 -> 0.143 | Helped |
| combine node, block, and graph granularities with hierarchical aggregation rather than putting MoEs together in one model (ablation) | HLSyn target kernels · Random fine-tuning split | MSE: 0.984 -> 0.452 | Helped |
| add block- and graph-level MoEs to the node-only hierarchy (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.164 -> 0.143 | Helped |
| add node- and graph-level MoEs to the block-only hierarchy (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.177 -> 0.143 | Helped |
| add node- and block-level MoEs to the graph-only hierarchy (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.279 -> 0.143 | Helped |
| use hidden representations rather than input features for high-level gating (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.149 -> 0.143 | Helped |
| remove alternating joint and individual expert training (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.143 -> 0.159 | Hurt |
| remove the warmup stage (ablation) | Hierarchical MoE · HLSyn target kernels | MSE: 0.143 -> 0.184 | Hurt |
| increase HARP hidden size from 64 to 128 (ablation) | HARP · HLSyn target kernels | MSE: 0.202 -> 0.240 | Hurt |
Surprising findings
- Applying MoE on multiple granularities in a single model (N+B+G) performs worse than using a single granularity, motivating the hierarchy.
- MAML-based initialization performs poorly (offline MSE 0.732) with many source kernels, possibly due to conflicting meta-gradient directions.
Limitations
- Performance on complex real-world kernels remains challenging for all models, as acknowledged by the authors.
- Hierarchical MoE has about 3.7x more parameters than HARP (1.33M vs 0.36M), and online results show high variance across kernels.
Experimental setup
Datasets: HLSyn Rosetta (3d-rendering, spam-filter)
Benchmarks: HLSyn Rosetta 3d-rendering and spam-filter
Metrics: MSE geometric mean speedup over AutoDSE
Baselines: HARP HARP+MAML ProgSG Node MoE Block MoE Graph MoE
Builds on: HARP GraphMETRO MAML ProGraML
Closest papers by meaning
- A Circuit Domain Generalization Framework for Efficient Logic Synthesis in Chip Design (ICML 2024 · similarity 0.84)
- Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures (NeurIPS 2025 · similarity 0.83)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.83)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.83)
- STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.83)
- LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.