Research map / Architectures and efficiency / RNNs and normalization

Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Neural-symbolic reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FGP, a generative pre-training method that trains any neural architecture encoder to reconstruct a random-vector-based "flow surrogate" simulating forward and backward information flow, avoiding slow flow-based structures.

Problem

Flow-based neural architecture encoders capture information flow well but are much slower than simple GNN encoders, and existing generative pre-training methods adapted from chemistry (masked operation prediction) provide unclear guidance for neural architectures.

Approach

Compute a flow surrogate per architecture by propagating random vectors through the DAG in forward (fp-messages) and backward (bp-messages) topological passes, then pre-train any encoder (even non-flow GNNs) to reconstruct this surrogate with an MSE loss, optionally combined with a zero-cost-proxy auxiliary loss.

Key results

• On ResGatedGCN with 1% training labels, FGP improves Precision@1% over N/A by up to +106.0% (NB-101) and outperforms baselines in 23 of 27 settings.
• FGP trained with ResGatedGCN improves over FlowerFormer without pre-training in both performance and total runtime, while FlowerFormer encoding is up to 94x slower than GIN.
• In NPENAS-based NAS on NAS-Bench-201, the predictor pre-trained with FGP finds the best top-1 architecture at every search step among compared pre-training methods.

Measured effects reported in the paper

Surprising findings

• Pre-training with random-vector-based surrogates (no learned parameters in the surrogate computation) provides large gains, and repeating the surrogate process over multiple rounds underperforms.
• FlowerFormer (flow-based) is up to 94x slower to encode than GIN, motivating simple encoders with pre-training.

Limitations

• Gains are smaller on NB-301 and for the flow-based FlowerFormer encoder (e.g., +3.1% Kendall's tau on NB-101).
• Pre-training uses the full search space including test architectures (transductive), though an extra train-only setting is reported.
• Evaluated mainly on cell-based CNN and NLP/graph NAS benchmarks with CIFAR-10-based labels; other domains only partially explored.

Experimental setup

Datasets: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 NAS-Bench-NLP NAS-Bench-Graph TIN DTD OxfordPet

Benchmarks: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 NAS-Bench-NLP NAS-Bench-Graph

Metrics: Kendall's tau Precision@1% Precision@5% Precision@10% top-1 test error in NAS

Baselines: N/A (supervised only) GraphCL Arch2vec GMAE ZC-Proxy FlowerFormer

Builds on: FlowerFormer NPENAS GMAE Arch2vec ZC-Proxy

Closest papers by meaning

• FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer (CVPR 2024 · similarity 0.86)
• Prior Knowledge Guided Neural Architecture Generation (ICML 2025 · similarity 0.85)
• Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · similarity 0.83)
• Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.83)
• Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.83)
• DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning (AAAI 2024 · similarity 0.83)