Research map / Optimization / Optimizers
MotherNet: Fast Training and Inference via Hyper-Network Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotherNet, a transformer hypernetwork meta-trained on synthetic tabular tasks that generates the weights of a small MLP child network in a single forward pass for in-context classification of tabular data without per-dataset gradient descent.
Problem
Existing tabular in-context learners like TabPFN are slow at inference and limited in training set size, while gradient-trained neural networks and tuned tree models require costly per-dataset training and hyperparameter tuning on small tabular classification datasets.
Approach
A TabPFN-style transformer is adapted as a hypernetwork: it reads the labeled training set and outputs low-rank weight components of a feed-forward child network that makes predictions directly, trained end-to-end on synthetic prior datasets with no dataset-specific gradient descent.
Key results
- On small CC-18 datasets, MotherNet reaches normalized AUC 0.780 with fit plus predict time of 0.14 s on GPU, versus 4.9 s for tuned XGBoost (1h HPO).
- MotherNet outperforms HyperFast (no FT normalized AUC 0.530; 1h tuned 0.729) and tuned MLP (0.744) on CC-18 test splits.
- On TabZilla, MotherNet has mean normalized AUC 0.84, comparable to SAINT and ResNet, but ranks below TabPFN and several tree-based methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate child-network weights in one forward pass without dataset-specific gradient descent | MotherNet · OpenML CC-18 small | normalized AUC: 0.744±0.025 -> 0.780±0.037 | Helped |
| use TabPFN's probabilistic predictions to train a dataset-specific distilled MLP | MLP-Distill · OpenML CC-18 small | normalized AUC: 0.744±0.025 -> 0.802±0.027 | Helped |
| remove ensembling and use a single network (ablation) | MotherNet · OpenML CC-18 small | normalized AUC: 0.780±0.037 -> 0.716±0.026 | Hurt |
| add dataset-specific gradient descent fine-tuning to the child model (ablation) | MotherNet · OpenML CC-18 small | normalized AUC: 0.716±0.026 -> 0.708±0.024 | Hurt |
| use HyperFast without per-dataset fine-tuning (ablation) | HyperFast · OpenML CC-18 small | normalized AUC: 0.681±0.020 -> 0.530±0.030 | Hurt |
| use low-rank child-network weights instead of full weights (ablation) | MotherNet · validation set | AUC | Helped |
| add one-hot encoding for classes in the prediction network (ablation) | MotherNet · tabular classification datasets | prediction performance | Helped |
| subsample to 3000 data points | MotherNet · TabZilla | normalized AUC: 0.84±0.18 | Mixed |
| replace in-context TabPFN predictions with the generated compact child network for prediction | MotherNet · OpenML CC-18 small | predict time (s): 0.337 -> 0.006 | Helped |
| use MotherNet instead of XGBoost for fit plus prediction | MotherNet · OpenML CC-18 small | fit + predict (s): 4.936 -> 0.143 | Helped |
| use MotherNet rather than TabPFN | MotherNet · OpenML CC-18 small | normalized AUC: 0.850±0.026 -> 0.780±0.037 | Hurt |
| use MotherNet rather than CatBoost | MotherNet · TabZilla | normalized AUC: 0.92±0.18 -> 0.84±0.18 | Hurt |
Surprising findings
- Fine-tuning MotherNet-generated networks with gradient descent did not improve results even with 1h of GPU hyperparameter search, suggesting the learned regularization is lost.
- MotherNet and TabPFN fail on synthetic discontinuous 1D functions and parity-like boolean functions where Random Forest does well.
Limitations
- Quadratic attention limits scaling; evaluated only on training sets of about 3000 samples or fewer for TabZilla.
- One-hot encoding of classes is needed for good child networks, requiring bagging over 8 ensemble members.
- Underperforms on discontinuous functions and boolean parity-style tasks, reflecting prior limitations.
Experimental setup
Datasets: OpenML CC-18 small subset TabZilla benchmark synthetic TabPFN prior datasets
Benchmarks: OpenML CC-18 (small, 30 datasets) TabZilla
Metrics: ROC AUC normalized AUC rank fit time predict time
Baselines: TabPFN HyperFast MLP-distill XGBoost Random Forest Logistic Regression KNN ResNet CatBoost SAINT
Builds on: TabPFN HyperFast Conditional Neural Processes XGBoost
Closest papers by meaning
- HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories (CVPR 2025 · similarity 0.85)
- HyperFast: Instant Classification for Tabular Data (AAAI 2024 · similarity 0.85)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.84)
- TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (NeurIPS 2024 · similarity 0.84)
- Nested Learning: The Illusion of Deep Learning Architectures (NeurIPS 2025 · similarity 0.84)
- Hypernetwork Approach to Bayesian MAML (Student Abstract) (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.