Research map / Graph learning / GNN architectures
NN-Former: Rethinking Graph Structure in Neural Architecture Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NN-Former, a transformer predictor for neural architecture DAGs whose attention and feed-forward layers inject adjacency and sibling topology, evaluated on accuracy (NAS-Bench-101/201) and latency (NNLQ) prediction.
Problem
Neural predictors for accuracy and latency must represent DAG architectures; GNNs lack expressiveness and global-attention transformers generalize poorly as architecture depth grows.
Approach
Adjacency-Sibling Multihead Attention applies four topology masks (forward adjacency, backward adjacency, shared-parent siblings, shared-child siblings) across heads, and a Bidirectional Graph Isomorphism FFN concatenates forward and backward graph convolutions, removing the need for position encoding.
Key results
- On NAS-Bench-101 with 1% training data, NN-Former reaches Kendall's Tau 0.877, above NAR-Former V2 (0.861).
- On NAS-Bench-201 with 1% training data, Kendall's Tau is 0.804 versus 0.752 for NAR-Former V2; it is not best at 10% (0.890 vs 0.901 for NAR-Former).
- In the out-of-domain NNLQ latency setting, average Acc(10%) is 74.31% versus 62.70% for NAR-Former V2, and in-domain average MAPE is 2.85% versus 3.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NN-Former instead of the strongest listed baseline | neural architecture accuracy prediction · NAS-Bench-101, 0.02% training | Kendall’s Tau: 0.666 -> 0.709 | Helped |
| use NN-Former instead of NAR-Former V2 | neural architecture accuracy prediction · NAS-Bench-201, 1% training | Kendall’s Tau: 0.752 -> 0.804 | Helped |
| use NN-Former instead of NAR-Former | neural architecture accuracy prediction · NAS-Bench-201, 10% training | Kendall’s Tau: 0.901 -> 0.890 | Hurt |
| use NN-Former instead of NAR-Former V2 | latency prediction · NNLQ, out-of-domain average | MAPE: 10.55 / 9.13 -> 8.39 / 7.96 | Helped |
| use NN-Former instead of NAR-Former V2 | latency prediction · NNLQ, out-of-domain average | Acc(10%): 62.70 / 67.40 -> 74.31 / 75.47 | Helped |
| use NN-Former instead of NAR-Former V2 | latency prediction · NNLQ, in-domain average | MAPE: 3.07 / 3.00 -> 2.85 / 2.65 | Helped |
| replace global attention with ASMA attention masks (ablation) | NN-Former latency predictor · NNLQ, NAS-Bench-201 out-of-domain | Acc(10%): 58.45% -> 69.90% | Helped |
| add ASMA while retaining vanilla FFN (ablation) | NN-Former accuracy predictor · NAS-Bench-101, 0.04% training | Kendall’s Tau: 0.4598 -> 0.6538 | Helped |
| add BGIFFN while retaining global attention (ablation) | NN-Former accuracy predictor · NAS-Bench-101, 0.04% training | Kendall’s Tau: 0.4598 -> 0.7656 | Helped |
| add NAR positional encoding (ablation) | NN-Former accuracy predictor with ASMA · NAS-Bench-101, 0.04% training | Kendall’s Tau: 0.7654 -> 0.7449 | Hurt |
| add Laplacian positional encoding (ablation) | NN-Former accuracy predictor with ASMA · NAS-Bench-101, 0.04% training | Kendall’s Tau: 0.7654 -> 0.7063 | Hurt |
| replace elementwise addition with Hadamard product in BGIFFN (ablation) | NN-Former accuracy predictor · NAS-Bench-101, 0.04% training | Kendall’s Tau: 0.7654 -> 0.7076 | Hurt |
Surprising findings
- Adding positional encodings (NAR PE or Laplacian) decreases performance of NN-Former, suggesting the topology masks make position encoding unnecessary.
- Sibling-node masks help on Cora citation prediction but not on ogbg-code2 AST prediction when added to DAG Transformer.
Limitations
- Does not reach the best result on NAS-Bench-201 at 10% training data (0.890 vs 0.901 for NAR-Former).
- Sibling-node benefit on general DAG tasks is inconsistent and the sibling mask was added without careful calibration.
Experimental setup
Datasets: NAS-Bench-101 NAS-Bench-201 NNLQ Cora ogbg-code2
Benchmarks: NAS-Bench-101 NAS-Bench-201 NNLQ
Metrics: Kendall's Tau MAPE Acc(10%)
Baselines: ReNAS NAO NP GATES GMAE-NAS Graphormer TNASP NAR-Former PINAT GraphTrans NAR-Former V2 NNLP
Builds on: NAR-Former V2 Graph Convolutional Network Graph Attention Network DAG Transformer TNASP
Closest papers by meaning
- A Topological Perspective on Demystifying GNN-Based Link Prediction Performance (ICLR 2024 · similarity 0.88)
- Directed Acyclic Graph Neural Networks (ICLR 2021 · similarity 0.88)
- A Generalization of ViT/MLP-Mixer to Graphs (ICML 2023 · similarity 0.88)
- A Closer Look at Graph Transformers: Cross-Aggregation and Beyond (NeurIPS 2025 · similarity 0.88)
- Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (ICLR 2020 · similarity 0.88)
- Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.