Research map / Architectures and efficiency / Attention and transformers

CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CRUMB, a training-free inference wrapper for PFNs that clusters test queries, selects a distributionally matched training context per cluster via greedy MMD (kernel herding), and runs batched PFN inference.

Problem

PFN in-context inference scales quadratically with the training context size, making prediction on very large tabular datasets prohibitive; existing context selection either loses locality or prevents batching of test queries.

Approach

Cluster the test queries (not the training data) and, for each test cluster, select a small training subset by greedily minimising MMD to the cluster via kernel herding with RFF and batched greedy selection, so all queries in a cluster share one context and can be batched.

Key results

• On TabICL-v2 across 51 TabArena datasets with n=0.1N, CRUMB average rank 2.975 vs per-query kNN 2.782 (not significant, p_adj=0.106) and MICP 3.304 (p_adj~1e-4), using K=20 forward passes instead of T=200 for kNN.
• Across TabICL-v1, TabICL-v2, TabPFN-v2, 38 classification datasets and 5 sampling proportions, CRUMB average rank 1.809 vs MICP 2.022 and Uniform 2.169.
• Under covariate drift, CRUMB's accuracy advantage over MICP grows from +4.9% (tau=0) to +17.1% (tau=1).

Surprising findings

• Under the ablation, clustering alone (Case B) gave no improvement over uniform subsampling; gains came from clustering combined with MMD retrieval.
• Centroid-NN and Voronoi-Uniform retrieval within the same test-clustering framework underperform MMD herding.

Limitations

• Requires the full set of test queries upfront; online version reduces to MICP-style routing with cached contexts.
• Euclidean k-means and isotropic Gaussian RBF kernel with median heuristic may be poorly suited to high-dimensional or mixed-type features.
• Inherits the ceiling of the underlying PFN; no fine-tuning explored.

Experimental setup

Datasets: TabArena Higgs

Benchmarks: TabArena Higgs Anneal Amazon Employee Access Airfoil Self Noise Diabetes130US APSFailure SDSS17 customer_satisfaction_in_airline GiveMeSomeCredit

Metrics: average rank accuracy RMSE R^2 Wilcoxon signed-rank with Bonferroni correction

Baselines: Full context per-query kNN MICP Uniform subsampling Centroid-NN Voronoi-Uniform

Builds on: TabPFNv2 TabICL MixturePFN (MICP) kernel herding random Fourier features

Closest papers by meaning

• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (NeurIPS 2024 · similarity 0.86)
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.85)
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.85)
• Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.84)
• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data (NeurIPS 2024 · similarity 0.84)
• Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.84)