Research map / Architectures and efficiency / Pruning and efficient tuning

HyperFast: Instant Classification for Tabular Data

AAAI 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HyperFast, a meta-trained hypernetwork that generates the weights of a small MLP for an unseen tabular dataset in a single forward pass, enabling instant classification without training.

Problem

Training and hyperparameter tuning of classifiers on tabular data is computationally expensive and slow, while existing pre-trained tabular approaches such as TabPFN are limited to small datasets (≤1000 training examples, 100 features, 10 classes).

Approach

A hypernetwork is meta-trained across many datasets; given a labeled support set, it uses random features, PCA, shared MLP modules, pooling over support samples, and nearest-neighbor biases to directly predict all weights of a main MLP, which then classifies query points in one forward pass.

Key results

• On the mini-test with a 5-minute budget, HyperFast achieves the best mean rank (3.547) and mean balanced accuracy (71.33%), above AutoGluon (69.94%, rank 4.273) and XGBoost (69.85%).
• HyperFast single forward pass predicts in under a second on large datasets, achieving overall best performance across runtime budgets from 1 second to 5 minutes on the big test.
• Ablation shows removing random features and PCA drops balanced accuracy by about 6 to 8 points (81.5% to 75.4% and 73.7%).

Measured effects reported in the paper

Surprising findings

• An MLP with the same architecture as the generated network (including RF and PCA transforms) performs on par with XGBoost when trained directly.
• Random initialization of the main network's linear layers instead of hypernetwork-generated weights causes the largest ablation drop (about 9.3 points).

Limitations

• Performance drops relative to AutoML systems at larger time budgets (up to 1h) and is only on par with them.
• Restricted to classification; regression and non-tabular modalities are left to future work.
• Hypernetwork is large (1.27B parameters) and meta-training overfits to meta-training datasets over long training.

Experimental setup

Datasets: OpenML-CC18 Dogs genomics UK Biobank HapMap3

Benchmarks: OpenML-CC18 meta-test subset HapMap3 subpopulation prediction UK Biobank phenotype prediction

Metrics: balanced accuracy average rank runtime

Baselines: XGBoost LightGBM CatBoost Auto-Sklearn 2.0 AutoGluon SAINT TabPFN NODE FT-Transformer T2G-Former DANet Net-DNF

Builds on: hypernetworks TabPFN Deep Sets random features Prototypical Networks

Closest papers by meaning

• TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.86)
• TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.86)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.85)
• TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.85)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (NeurIPS 2024 · similarity 0.85)
• TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.85)