Research map / Architectures and efficiency / Attention and transformers
PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with multi-granularity pooling | Transformer token mixing · Long Range Arena | AVG. accuracy: 58.77 -> 61.05 | Helped |
| Replace self-attention with multi-granularity pooling | Transformer token mixing · LRA Text classification, sequence length 4096 | training speed: 1.8 -> 16.2 steps/s | Helped |
| Replace self-attention with multi-granularity pooling | Transformer token mixing · GLUE validation | AVG.: 73.46 -> 76.80 | Helped |
| Remove the second-stage global aggregation (ablation) | PoNet · STS-B validation | Spearman correlation: 84.57 -> 78.38 | Hurt |
| Remove global aggregation (ablation) | PoNet · STS-B validation | Spearman correlation: 84.57 -> 64.61 | Hurt |
| Remove segment max-pooling (ablation) | PoNet · CoLA validation | Matthews correlation: 45.36 -> 44.21 | Hurt |
| Remove local max-pooling (ablation) | PoNet · CoLA validation | Matthews correlation: 45.36 -> 41.44 | Hurt |
| Remove segment and local max-pooling (ablation) | PoNet · CoLA validation | Matthews correlation: 45.36 -> 11.36 | Hurt |
| Replace SSO with NSP during pre-training (ablation) | PoNet · STS-B validation | Spearman correlation: 84.57 -> 75.32 | Hurt |
| Remove the sentence-structure pre-training objective (ablation) | PoNet · STS-B validation | Spearman correlation: 84.57 -> 69.83 | Hurt |
| Remove the second-stage global aggregation (ablation) | PoNet · CoLA validation | Matthews correlation: 45.36 -> 46.18 | Helped |
| Remove global aggregation (ablation) | PoNet · CoLA validation | Matthews correlation: 45.36 -> 49.51 | Helped |
Closest papers by meaning
- Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models (NeurIPS 2025 · similarity 0.88)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.86)
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.86)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.86)
- FFN Fusion: Rethinking Sequential Computation in Large Language Models (NeurIPS 2025 · similarity 0.86)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.