Research map / Architectures and efficiency / RNNs and normalization
BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiSHop, a tabular deep learning model built from generalized sparse modern Hopfield (GSH) layers applied column-wise and row-wise in stacked encoder-decoder blocks, outperforming tree-based and deep tabular baselines with far fewer HPO runs.
Problem
Deep learning models for tabular data lag behind tree-based methods due to non-rotationally invariant data structure and feature sparsity.
Approach
Replace softmax attention with a learnable-sparsity generalized sparse modern Hopfield (alpha-entmax) layer and alternate column-wise (per-feature across embedding patches) and row-wise (pooling across features) Hopfield blocks in a hierarchical encoder-decoder, so each scale has its own sparsity.
Key results
- On Dataset I, BiSHop averages roughly 91.9 AUC in its ablation table and wins on most datasets, e.g., 88.49 on Blastchar and 91.88 on SeismicBumps versus best baselines around 84-87 and 81.6.
- On the 19-dataset OpenML benchmark, BiSHop achieves mean score 81.21 vs. 80.65 for XGBoost and 80.48 for GBDT, with mean rank 2.79 and 11 optimal and 8 near-optimal results.
- BiSHop uses less than 10% (on average) of the HPO runs used by the benchmark baselines (400 HPOs each).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add categorical embedding (ablation) | BiSHop · Datasets I average | AUC (%): 90.82 -> 91.86 | Helped |
| add numerical embedding (ablation) | BiSHop · Datasets I average | AUC (%): 88.62 -> 91.86 | Helped |
| add patch embedding (ablation) | BiSHop · Datasets I average | AUC (%): 90.06 -> 91.86 | Helped |
| add decoder (ablation) | BiSHop · Datasets I average | AUC (%): 89.16 -> 91.86 | Helped |
| add BiSHopModule (ablation) | BiSHop · Datasets I average | AUC (%): 89.17 -> 91.86 | Helped |
| switch row-wise and column-wise directions (ablation) | BiSHop · Datasets I | AUC (%): <1% | No clear effect |
| rotate the datasets (ablation) | BiSHop · Datasets I | AUC (%) | No clear effect |
| stack 4 BiSHopModule layers instead of other tested depths (ablation) | BiSHop · Datasets I | AUC (%) | Helped |
| use generalized sparse Hopfield model instead of dense Hopfield or attention (ablation) | BiSHop Hopfield layers · tabular datasets | AUC | Helped |
| use generalized sparse rather than dense Hopfield retrieval (ablation) | Hopfield retrieval · memory retrieval | convergence rate | Helped |
| remove features in increasing or decreasing feature-importance order (ablation) | BiSHop · Datasets I | AUC | Mixed |
Surprising findings
- Removing the BiSHopModule in the component ablation (w/o BiSHopModule) gives an average AUC close to removing the decoder, suggesting several components matter similarly.
- Rotating the datasets and swapping row/column directions causes only marginal (<1%) performance drops, indicating robustness to rotation.
Limitations
- Does not use external memory capabilities of modern Hopfield models, as noted by the authors.
- Sparse alpha-entmax retains O(n^2) space complexity in the forward pass, so sparsity does not reduce compute.
- Experiments use only truncated datasets (up to 10,000 training samples) due to limited compute; Dataset II used one-third random subsets.
Experimental setup
Datasets: Adult Bank Blastchar Income SeismicBumps Shrutime Spambase Qsar Jannis OpenML CC/NC/CR/NR suites (19 datasets)
Benchmarks: Adult Bank Blastchar Income SeismicBumps Shrutime Spambase Qsar Jannis OpenML tabular benchmark (Grinsztajn et al. CC/NC/CR/NR suites)
Metrics: AUC accuracy R^2 number of HPO runs
Baselines: MLP TabNet TabTransformer FT-Transformer SAINT TabPFN TANGOS T2G-FORMER LightGBM CatBoost XGBoost ResNet
Builds on: modern Hopfield model generalized sparse modern Hopfield model (STanHop) FT-Transformer SAINT TabPFN
Closest papers by meaning
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.86)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.86)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.86)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.86)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.86)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.