Research map / Architectures and efficiency / Attention and transformers
Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames transformer pooling as signal-optimal vector quantization and shows attention-based adaptive pooling (AdaPool) with a signal query is robust to varying signal-to-noise ratios across synthetic, MPE, BoxWorld, and CIFAR tasks.
Problem
Standard transformer output pooling (AvgPool, MaxPool, ClsToken) degrades unpredictably when the fraction of task-relevant (signal) input vectors versus distractor (noise) vectors changes, which is common in RL and vision.
Approach
Treat global pooling as a vector quantizer that minimizes signal loss, show AvgPool and MaxPool are special cases of attention pooling, and propose using a known signal vector (e.g., the controlled entity's token) as the attention query, with error bounds on how closely it approximates the signal-optimal centroid.
Key results
- On the synthetic KNN-Centroid task at SNR 0.01, AdaPool signal loss is 0.037 versus 0.066 (MaxPool), 0.093 (AvgPool), and 0.102 (ClsToken).
- In simple tag with increasing obstacles, final reward drops were AvgPool 77.4%, ClsToken 70.4%, MaxPool 60.7%, and AdaPool 50.9%.
- On CIFAR-10 and CIFAR-100, AdaPool Focal/Mean queries reach about 87.8-88.0% and 61.2% top-1, outperforming ClsToken (84.52 / 55.56).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive pooling instead of MaxPool | 12-layer transformer · KNN-Centroid, SNR = 0.01 (KNN-1) | signal loss (MSE): 0.066 ±0.102 -> 0.037 ±0.006 | Helped |
| Use adaptive pooling instead of AvgPool | 12-layer transformer · KNN-Centroid, SNR = 0.25 (KNN-32) | signal loss (MSE): 0.016 ±0.008 -> 0.002 ±0.000 | Helped |
| Use adaptive pooling instead of AvgPool | 12-layer transformer · KNN-Centroid, SNR = 1.00 (KNN-128) | signal loss (MSE): 0.000 ±0.000 -> 0.000 ±0.000 | No clear effect |
| Use the four central patch embeddings as the AdaPool query rather than a corner patch (ablation) | ViT · CIFAR-10 test | Top-1 accuracy: 87.00 ±0.30 -> 87.98 ±0.42 | Helped |
| Use the mean of all patch embeddings as the AdaPool query rather than a corner patch (ablation) | ViT · CIFAR-10 test | Top-1 accuracy: 87.00 ±0.30 -> 87.84 ±0.30 | Helped |
| Use the four central patch embeddings as the AdaPool query rather than a corner patch (ablation) | ViT · CIFAR-100 test | Top-1 accuracy: 57.08 ±0.31 -> 61.22 ±0.33 | Helped |
| Use the mean of all patch embeddings as the AdaPool query rather than a corner patch (ablation) | ViT · CIFAR-100 test | Top-1 accuracy: 57.08 ±0.31 -> 61.23 ±0.20 | Helped |
| Use adaptive pooling with a focal or mean query instead of ClsToken | ViT · CIFAR-10 test | Top-1 accuracy: 84.52 ±0.21 -> 87.98 ±0.42 | Helped |
| Use adaptive pooling with a mean query instead of ClsToken | ViT · CIFAR-100 test | Top-1 accuracy: 55.56 ±0.13 -> 61.23 ±0.20 | Helped |
| Increase the noise volume and ratio in simple tag observations | MPE simple tag · Simple Tag + Noise | final mean reward decline: AdaPool drops 50.9%; MaxPool falls 60.7%; ClsToken drops 70.4%; AvgPool drops 77.4%. | Mixed |
| Use adaptive pooling instead of MaxPool as observations expand from entities to all pixels | BoxWorld · Entities only (8) vs all pixels (50) | performance | Mixed |
| Use a corner patch rather than central patches or the mean of all patches as the AdaPool query (ablation) | ViT · CIFAR-10 and CIFAR-100 | Top-1 accuracy | Hurt |
Surprising findings
- MaxPool performed worst by a wide margin in the MPE simple centroid task, even when signal was sparse.
- A transformer with AvgPool or MaxPool can fail to approximate the MAX target, while AdaPool approximates min and mean targets better; AdaPool also cannot perfectly fit max targets.
Limitations
- AdaPool requires choosing a query that is known to be signal; poor queries (e.g., Corner) degrade performance.
- Multi-head proofs and the MaxPool special case rely on extreme weight-matrix values that gradient descent rarely reaches in practice.
Experimental setup
Datasets: synthetic KNN-centroid dataset (1M samples) MPE simple tag MPE simple centroid (custom) BoxWorld CIFAR-10 CIFAR-100
Benchmarks: synthetic KNN-Centroid task MPE simple centroid MPE simple tag BoxWorld CIFAR-10 CIFAR-100
Metrics: signal loss (MSE) mean episode reward Top-1 accuracy
Baselines: AvgPool MaxPool ClsToken Centroid baseline Target baseline
Builds on: Hopfield Pooling / attention as associative memory AdaPool (Stergiou & Poppe) Vision Transformer PPO Multi-Particle Environment
Closest papers by meaning
- Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models (NeurIPS 2025 · similarity 0.87)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.84)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.84)
- Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference (AAAI 2025 · similarity 0.84)
- Unified Noise Steering for Efficient Human-Guided VLA Adaptation (NeurIPS 2026 · similarity 0.84)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.