Research map / Architectures and efficiency / Attention and transformers
Polynomial-based Self-Attention for Table Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CheAtt, a Chebyshev-polynomial-based self-attention layer that mitigates oversmoothing in Transformer table representation models, and plugs it into TabTransformer, SAINT, and MET to improve tabular classification and regression.
Problem
Transformers for tabular data suffer from oversmoothing in self-attention, where deeper layers produce similar token representations and attention acts as a low-pass filter, limiting representation quality.
Approach
Replace standard self-attention (a first-order graph filter A V) with a Chebyshev matrix polynomial sum of alpha_k T_k(A) V, with learnable coefficients; the low order is justified by PageRank-style convergence of attention matrices and the small token count of tables.
Key results
- Adding CheAtt raises TabTransformer's average score from 77.5% to 84.2% (8.65% relative improvement), with SAINT 84.5 to 85.1 and MET 79.4 to 83.1.
- TabTransformer+CheAtt, SAINT+CheAtt, and MET+CheAtt are reported to outperform all baselines on 6 of 10 datasets.
- Sensitivity over polynomial order k in {2,3,5,10} shows performance saturates after a small k, with best results mostly at k<=5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with CheAtt in TabTransformer | TabTransformer · All datasets (average) | score (%): 77.5 -> 84.2; 8.65% | Helped |
| Replace self-attention with CheAtt in SAINT | SAINT · All datasets (average) | score (%): 84.5 -> 85.1; 0.64% | Helped |
| Replace self-attention with CheAtt in MET | MET · All datasets (average) | score (%): 79.4 -> 83.1; 4.66% | Helped |
| Vary Chebyshev polynomial order among 2, 3, 5, and 10 (ablation) | TabTransformer+CheAtt, SAINT+CheAtt, and MET+CheAtt · Alphabank, Contraceptive, Medicalcost | AUROC: 61.3; 61.5; 61.6; 61.5; 61.4; 61.3; 62.0; 61.6; 62.2; 60.9; 61.4; 61.3; 75.9; 75.7; 76.5; 75.0; 75.2; 75.1; 77.1; 75.6; | Mixed |
| Use Chebyshev rather than Power, Legendre, or Jacobi polynomial basis (ablation) | TabTransformer+CheAtt, SAINT+CheAtt, and MET+CheAtt · Default, Buddy, Superconductivity | AUROC / R²: Default: Chebyshev 78.9, 78.4, 77.8; Buddy: 91.8, 94.9, 85.5; Superconductivity: 87.6, 87.5, 88.2 | Mixed |
Surprising findings
- Legendre polynomial basis performs well for SAINT+CheAtt on some datasets, and the MET+CheAtt best order is k=2 rather than 5 as for the others.
- Gradient-boosted trees (XGBoost) and Random Forest remain best on Activity, beating all Transformer-based variants.
Limitations
- Relies on the assumption that attention matrices are fully connected and aperiodic, which the paper argues but does not fully verify; the convergence argument is heuristic.
- Tuning of polynomial order k and hyperparameters is per dataset, and gains on SAINT are small (0.64%).
- Scalability argument is limited to tables with few columns; not applicable to large token counts like images or graphs.
Experimental setup
Datasets: Income Default Phishing Alphabank Clave Contraceptive Activity Buddy Medicalcost Superconductivity
Benchmarks: Income Default Phishing Alphabank Clave Contraceptive Activity Buddy Medicalcost Superconductivity
Metrics: AUROC R^2
Baselines: MLP Decision Tree Logistic/Linear Regression XGBoost Random Forest VIME TabNet TabTransformer SAINT MET
Builds on: TabTransformer SAINT MET PageRank Chebyshev polynomial graph filters
Closest papers by meaning
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.87)
- Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data (AAAI 2025 · similarity 0.86)
- Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.85)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.85)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.85)
- PTaRL: Prototype-based Tabular Representation Learning via Space Calibration (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.