Research map / Architectures and efficiency / Attention and transformers
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
Jannik Kossen, Neil Band, Clare Lyle, Aidan Gomez, Tom Rainforth, Yarin Gal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention between datapoints to look up targets from matching duplicate rows | NPT · Protein semi-synthetic lookup | RMSE: 3.62 -> 0.44 | Improved |
| Randomize selected features and add one to duplicate targets, requiring learned feature selection and target adjustment | NPT · Random Feats. + Add One | Test RMSE: 0.75 vs 7.80 for 1-NN | Improved |
| Corrupt information from other datapoints at test time | NPT · CIFAR-10 | accuracy: -1.2 | Hurt |
| Corrupt information from other datapoints at test time | NPT · Protein | RMSE: -21% | Hurt |
| Corrupt information from other datapoints at test time | NPT · Breast Cancer | accuracy: 0.0 | No clear effect |
| Corrupt information from other datapoints at test time | NPT · Forest, Kick, and Breast Cancer | accuracy: Forest -0.1; Kick -0.1; Breast Cancer 0.0 | Mixed |
| Add ABA layers after ABD layers (ablation) | NPT · UCI tabular benchmarks | performance | Improved |
| Add stochastic feature masking during training (ablation) | NPT · UCI tabular benchmarks | performance | Improved |
Closest papers by meaning
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.84)
- Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.83)
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.83)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.82)
- On Understanding Attention-Based In-Context Learning for Categorical Data (ICML 2025 · similarity 0.82)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
