Research map / Graph learning / GNN architectures
FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLOWERFORMER, a graph transformer for neural architecture encoding that adds asynchronous forward/backward message passing and flow-masked global attention to predict architecture performance.
Problem
Predicting neural architecture performance without full training requires accurate architecture representations; existing graph message passing and graph transformers do not capture the forward and backward information flows of neural architectures.
Approach
Treats architectures as DAGs and combines a flow encoder that runs asynchronous forward then backward message passing in topological generation order with a global attention module masked to node pairs lying on a common information-flow path.
Key results
- Kendall's Tau on NAS-Bench-101 at 50% training data: FLOWERFORMER 89.6 vs TA-GATES 86.3 and GatedGCN 84.8 (average rank 1.00 across 12 settings in Table 2).
- Ranks best in Kendall's Tau in every computer vision setting; Precision@K best in 10 of 12 settings.
- On NAS-Bench-Graph and NAS-Bench-ASR, FLOWERFORMER achieves best Kendall's Tau in most settings (e.g., NB-ASR 52.2 vs 50.9 for TA-GATES at 50%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the flow encode module (ablation) | FLOWERFORMER · NAS-Bench-101 | Kendall's Tau (×100): 42.5 (1.6) -> 86.1 (0.8) | Helped |
| add asynchronous message passing (ablation) | FLOWERFORMER · NAS-Bench-101 | Kendall's Tau (×100): 56.8 (5.0) -> 86.1 (0.8) | Helped |
| add forward-backward message passing (ablation) | FLOWERFORMER · NAS-Bench-101 | Kendall's Tau (×100): 83.9 (2.6) -> 86.1 (0.8) | Helped |
| add flow-aware global attention (ablation) | FLOWERFORMER · NAS-Bench-101 | Kendall's Tau (×100): 83.2 (3.9) -> 86.1 (0.8) | Helped |
| add forward-backward message passing (ablation) | FLOWERFORMER · NAS-Bench-201 | Kendall's Tau (×100): 88.6 (0.8) -> 89.8 (0.3) | Helped |
| add flow-aware global attention (ablation) | FLOWERFORMER · NAS-Bench-201 | Kendall's Tau (×100): 89.5 (0.5) -> 89.8 (0.3) | No clear effect |
| add asynchronous message passing (ablation) | FLOWERFORMER · NAS-Bench-201 | Kendall's Tau (×100): 85.6 (0.7) -> 89.8 (0.3) | Helped |
| use FLOWERFORMER instead of DAGNN | NAS-Bench-Graph · NAS-Bench-Graph | Kendall's Tau (×100): 73.1 (0.8) -> 72.7 (0.2) | Hurt |
| use FLOWERFORMER instead of TA-GATES | NAS-Bench-ASR · NAS-Bench-ASR | Kendall's Tau (×100): 50.9 (0.8) -> 52.2 (1.4) | Helped |
Surprising findings
- GraphGPS, a graph transformer without flow modeling, performs notably worse than FLOWERFORMER, indicating plain graph transformers are insufficient for architecture encoding.
- TA-GATES performs poorly on NAS-Bench-Graph (about 33-36 Kendall's Tau), where message passing between nodes is needed.
Limitations
- Evaluated only on performance prediction tasks over NAS benchmark datasets.
- Improvements over TA-GATES on NAS-Bench-Graph and NAS-Bench-ASR are not uniform across all training portions.
Experimental setup
Datasets: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 NAS-Bench-ASR NAS-Bench-Graph
Benchmarks: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 NAS-Bench-ASR NAS-Bench-Graph
Metrics: Kendall's Tau Precision@K
Baselines: GatedGCN DAGNN GraphGPS DAGFormer NAR-Former TA-GATES
Builds on: TA-GATES DAGFormer GraphGPS DAGNN GATES
Closest papers by meaning
- Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding (NeurIPS 2025 · similarity 0.86)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.84)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.84)
- Wavy Transformer (NeurIPS 2025 · similarity 0.84)
- DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging (NeurIPS 2024 · similarity 0.84)
- Transformers need glasses! Information over-squashing in language tasks (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.