Research map / Architectures and efficiency / Attention and transformers
Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiXT, a linear-scaling Transformer that replaces Perceiver's iterative cross-attention with a bi-directional cross-attention where latents and tokens attend to each other using one shared similarity matrix.
Problem
Quadratic cost of full self-attention limits Transformers on long inputs, while Perceiver-style latent bottleneck architectures scale linearly but lag in performance and efficiency.
Approach
Latents and input tokens refine each other simultaneously in every layer via bi-directional cross-attention, exploiting an approximately symmetric attention pattern that lets one similarity matrix (computed once, using shared reference projections) serve both row- and column-wise softmax, cutting projections by about one third and keeping cost linear in input length.
Key results
- BiXT-Ti/4 reaches 83.1% top-1 on ImageNet1K at 15M parameters and no vision-specific internals, and BiXT-Ti/16 at 3.6G FLOPs reaches 81.8% versus DeiT3-S's 81.4% at 4.6G FLOPs.
- On LRA, BiXT matches the full Transformer accuracy (82.46% vs 82.34% on AAN; 39.42% vs 39.10% on Long ListOps) while using 25-28% fewer FLOPs and up to 8.4x higher throughput.
- Bi-directional cross-attention at matched FLOPs outperforms sequential cross-attention and all iterative Perceiver-style variants on ImageNet1K (e.g. 73.86% vs 73.10% for the sequential d11 variant and about 56-61% for iterative variants).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sequential cross-attention with bi-directional cross-attention at 12 layers (ablation) | BiXT · ImageNet1K, 120 epochs | Top-1 accuracy: 73.79{\scriptstyle~\pm~0.32} -> 73.86{\scriptstyle~\pm~0.39} | No clear effect |
| Use bi-directional rather than sequential cross-attention with one additional layer at similar FLOPs (ablation) | BiXT · ImageNet1K, 120 epochs | Top-1 accuracy: 73.10{\scriptstyle~\pm~0.53} -> 74.10{\scriptstyle~\pm~0.14} | Helped |
| Replace Perceiver-style iterative attention with bi-directional cross-attention (ablation) | BiXT · ImageNet1K, 120 epochs | Top-1 accuracy: 60.61{\scriptstyle~\pm~1.11} -> 74.10{\scriptstyle~\pm~0.14} | Helped |
| Add affine grouping to BiXT (ablation) | BiXT · ModelNet40 | OA | Helped |
| Add hierarchical point reduction to affine grouping (ablation) | BiXT · ModelNet40 | OA | Helped |
| Add local token refinement (ablation) | BiXT · ADE20K | mIoU | Helped |
| Add modality-specific token refinement and decoder (ablation) | BiXT · ShapeNetPart | class mIoU: 83.5% -> 84.7% | Helped |
| Increase image feature resolution using smaller patches (ablation) | BiXT-Ti · ImageNet1K | Top-1 accuracy: 80.1 -> 82.7 | Helped |
| Increase input resolution from 224² to 384² (ablation) | BiXT · ImageNet1K | Top-1 accuracy: +2.1% | Helped |
| Increase the number of latent vectors (ablation) | BiXT · ImageNet1K | Accuracy | Helped |
| Increase embedding dimension (ablation) | BiXT · ImageNet1K | Accuracy | Helped |
Surprising findings
- Attention maps from latents to tokens and from tokens to latents are approximately symmetric when trained with sequential cross-attention, motivating the shared-similarity design.
- Bi-directional attention offers a net accuracy gain at matched FLOPs, with the saved compute spent on an extra layer, and gives smaller seed variance.
Limitations
- Deep variants may struggle with extremely long sequences under tight compute and memory, since token depth is not decoupled from input length.
- Encoder-based design; causal masking for autoregressive generation is not directly applicable.
- Larger model scales were not explored due to compute and hyperparameter-search costs.
Experimental setup
Datasets: ImageNet1K ModelNet40 ShapeNetPart ADE20K ListOps (LRA) AAN (LRA)
Benchmarks: ImageNet1K ModelNet40 ShapeNetPart ADE20K Long ListOps AAN document retrieval
Metrics: top-1 accuracy overall accuracy mIoU FLOPs throughput (samples/s) memory
Baselines: Perceiver Perceiver v2 Perceiver-IO DeiT DeiT3 PointNet PointMLP PVTv2 BiFormer Transformer on LRA
Builds on: Perceiver Perceiver-IO DeiT PointMLP Long Range Arena
Closest papers by meaning
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.85)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (NeurIPS 2024 · similarity 0.85)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.84)
- Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models (NeurIPS 2025 · similarity 0.84)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.