Research map / Architectures and efficiency / Attention and transformers
Efficient Representation Learning via Adaptive Context Pooling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive ContextPool to the Base transformer | Transformer Base · WMT 2014 EN-DE | BLEU: 28.52 -> 28.91 | Helped |
| Add adaptive ContextPool to the Base transformer | Transformer Base · WMT 2014 EN-FR | BLEU: 39.19 -> 39.36 | Helped |
| Add ContextPool with 14 layers | Transformer · text8 test | BPC: 1.10 -> 1.07 | Helped |
| Add ContextPool with 14 layers | Transformer · enwik8 test | BPC: 1.00 -> 0.97 | Helped |
| Add ContextPool | ViT-B/16 · ImageNet-1K validation, 384² | Top1: 77.9 -> 79.2 | Helped |
| Add ContextPool | Swin-B · ImageNet-1K validation, 384² | Top1: 84.5 -> 85.6 | Helped |
| Add ContextPool | ResNet-50 · ImageNet-1K validation | Top-1: 76.5 -> 77.3 | Helped |
| Replace learned pooling weights with uniform weights (ablation) | ContextPool transformer · WMT 2014 EN-DE | BLEU: 28.91 -> 28.52 | Hurt |
| Replace learned pooling weights with uniform weights (ablation) | ContextPool ViT-B/16 · ImageNet-1K classification | Top1: 79.9 -> 78.9 | Hurt |
| Remove the locality prior (ablation) | ContextPool transformer · WMT 2014 EN-DE | BLEU: 28.91 -> 28.31 | Hurt |
| Remove the locality prior (ablation) | ContextPool ViT-B/16 · ImageNet-1K classification | Top1: 79.9 -> 78.3 | Hurt |
| Use a fixed window rather than an adaptive locality window (ablation) | ContextPool transformer · WMT 2014 EN-DE | BLEU: 28.91 -> 28.55 | Hurt |
Closest papers by meaning
- Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models (NeurIPS 2025 · similarity 0.90)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.87)
- Scale-invariant attention (NeurIPS 2025 · similarity 0.86)
- Contextures: Representations from Contexts (ICML 2025 · similarity 0.86)
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.85)
- Dual Dimensionality for Local and Global Attention (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.