Research map / Architectures and efficiency / Attention and transformers

Benchmarking Attention for Tabular Foundation Models

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks attention backends (SDPA efficient/cuDNN, FlashAttention-2/3/4, vLLM, SageAttention) for row and column attention in tabular foundation models across A100, H100, and B200 GPUs, with a roofline analysis of the .contiguous() copy overhead and a preliminary agent-driven FA4 optimization.

Problem

Efficient attention kernels are tuned for long 1D language-model sequences, while tabular in-context learners alternate row and column attention over 2D embedding grids with asymmetric, short-or-long sequences, small head dimensions, and strided layouts that force copies. Which backend is best for e

Approach

A reproducible kernel-level benchmark of tabular 2D attention patterns (row-first layout, B=1, H=12, D=64 main config) that measures forward and backward throughput and peak memory across backends and GPUs, combined with a roofline and copy-overhead model explaining why FA3/FA4 win row attention (no copies) while cuDNN wins long column attention.

Key results

• FA3 achieves up to 3.5x speedup over SDPA baseline and around 1.5x over cuDNN for row attention on A100/H100 at R>=128.
• cuDNN is state-of-the-art or close to it for column attention across GPUs, while FA4 leads column attention up to 256 columns on B200.
• End-to-end TabPFN-v2.6 with a mixed backend dispatch yields a mean 7.4% speedup over default dispatch on TabArena-lite with max difference of 0.004 in predictive score.

Surprising findings

• SageAttention drops ConTextTab accuracy by about 3 pp (significant) while TabPFN shows no significant change.
• FA3 performs slightly worse than FA2 for column attention up to 256 columns on H100.

Limitations

• Isolated kernels on synthetic tensors with B=1 and only bfloat16 and non-causal attention; does not capture surrounding layers or multi-GPU effects.
• End-to-end validation limited to TabPFN-v2.6 on a single GPU; TabICL-style multi-stage architectures are not covered.
• SageAttention results only on H100 and only for two models.

Experimental setup

Datasets: CARTE TabArena-lite TabArena TALENT

Benchmarks: TabArena-lite CARTE

Metrics: TFLOPS latency peak memory accuracy R^2 Elo

Baselines: SDPA (efficient) SDPA (cuDNN) FlashAttention-2 FlashAttention-3 FlashAttention-4 vLLM SageAttention

Builds on: FlashAttention scaled dot-product attention SageAttention vLLM

Closest papers by meaning

• TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.90)
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.89)
• Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.89)
• Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.89)
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
• Enhancing Tabular Learners with Context-Aware Semantic Embeddings (NeurIPS 2026 · similarity 0.89)