Research map / Architectures and efficiency / Attention and transformers
Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMFormer, a transformer variant with parallel additive and multiplicative (log-space) attention plus prompt-token queries, arguing arithmetic feature interaction is necessary for deep tabular learning.
Problem
Identifying an effective inductive bias for deep learning on tabular data, where standard transformers struggle to capture arithmetic (multiplicative) feature interactions and self-attention scales quadratically with feature count.
Approach
Adds a parallel multiplicative attention stream (attention in log-space followed by exponentiation) alongside additive attention, fuses candidates with an FC layer over the candidate dimension, uses top-k sparse attention, and replaces queries with learnable prompt tokens for linear complexity.
Key results
- On the four real-world datasets, AMF-F ranks first on average (rank 1.3±0.5) and AMF-A second (2.0±0.8), ahead of XGBoost (5.3±2.6) and the best baseline DCAP (4.3±1.7).
- On synthetic data AMFormer outperforms vanilla Transformer by up to +57% in fine-grained classification accuracy, +16% in data efficiency and +20% in generalization (as reported in the introduction).
- Prompt-based optimization cuts training time for AMF-F from 20.88 h to 3.11 h (about 7x faster) and GPU memory from 38 GB to 18 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the AMFormer module to AutoInt | AutoInt · EP | accuracy: +1.23% | Helped |
| Add the AMFormer module to AutoInt | AutoInt · CO | accuracy: +4.96% | Helped |
| Add the AMFormer module to transformer backbones | AutoInt and FT-Transformer · HC | AUC | Mixed |
| Add multiplicative attention alongside additive attention (ablation) | AMFormer · EP and MI | accuracy and MSE | Helped |
| Add prompt optimization (ablation) | AMFormer · EP and MI | accuracy and MSE | Helped |
| Replace additive attention with multiplicative attention (ablation) | AutoInt-based AMFormer · EP and MI | accuracy and MSE | Helped |
| Replace additive attention with multiplicative attention (ablation) | FT-Transformer-based AMFormer · EP | accuracy | Hurt |
| Use multiplicative-only rather than additive-only attention (ablation) | FT-Transformer-based AMFormer · MI | MSE | Helped |
| Vary prompt-token count (ablation) | AMF-F · EP | accuracy | Mixed |
| Increase the number of layers (ablation) | AMFormer · EP | accuracy | Mixed |
| Increase the number of layers (ablation) | AMFormer · CO | accuracy | Helped |
| Increase top-k interaction parameter (ablation) | AMFormer · EP and CO | accuracy | Mixed |
Surprising findings
- Multiplicative-only attention outperforms additive-only attention in three of four ablation cases.
- XGBoost is not outperformed by any deep baseline on all datasets, and DCN-V2 performs worst among the tested methods.
Limitations
- Core evidence for necessity of arithmetic interaction rests on a single custom synthetic dataset designed around the hypothesis.
- Hyperparameters such as top-k, prompt count, and layer number are tuned on the test-reported datasets with sensitivity analyses only.
- Some ablation results (e.g., FT-Transformer multiplicative-only on EP at 50.05) look anomalous and are not explained.
Experimental setup
Datasets: Epsilon (EP) Home Credit Default Risk (HC) Covtype (CO) MSLR-WEB10K (MI) synthetic arithmetic-interaction dataset
Benchmarks: Epsilon Home Credit Default Risk Covtype MSLR-WEB10K synthetic arithmetic dataset
Metrics: accuracy AUC MSE average rank
Baselines: XGBoost NODE DCN-V2 DCAP AutoInt FT-Transformer
Builds on: FT-Transformer AutoInt transformer self-attention
Closest papers by meaning
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.87)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.84)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.84)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.84)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.84)
- APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuning (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.