Research map / Architectures and efficiency / Attention and transformers
xT: Nested Tokenization for Larger Context in Large Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes xT, a two-stage nested-tokenization framework that lets pretrained vision backbones encode large images region-by-region and aggregates global context with a linear-attention or Mamba/Transformer-XL context encoder.
Problem
Vision pipelines handle large images only by down-sampling or cropping, losing high-frequency detail or global context; full-image end-to-end modeling of very large images runs out of GPU memory.
Approach
Independently encode regions of a large image with a small-image vision backbone (region encoder), then stream the resulting region features through a lightweight long-context sequence model (HyperAttention LLaMA-style, Mamba, or Transformer-XL with recurrence) so each region's features are contextualized globally with near-linear memory.
Key results
- On iNaturalist Reptilia, Swin-L xT XL reaches 73.47% top-1 versus 68.78% for Swin-L at 256 px, and xT variants improve accuracy by up to 8.6%.
- On xView3-SAR, xT variants improve overall F1 by up to 13.4 points and close-to-shore F1 by up to 6.0 points over the non-context Swin baselines.
- On Cityscapes detection, Swin-B-DetINO runs out of memory at 2048 input while the xT XL variant reaches 43.0 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add nested tokenization and Mamba context to Swin-T | Swin-T · iNaturalist 2018 Reptilia | top-1 accuracy: 53.76 -> 61.97 | Helped |
| Add nested tokenization and Hyper context to Swin-S | Swin-S · iNaturalist 2018 Reptilia | top-1 accuracy: 58.45 -> 63.62 | Helped |
| Add nested tokenization and Hyper context to Hiera-B | Hiera-B · iNaturalist 2018 Reptilia | top-1 accuracy: 48.60 -> 57.20 | Helped |
| Add nested tokenization and Hyper context to Swin-B | Swin-B · iNaturalist 2018 Reptilia | top-1 accuracy: 58.57 -> 64.08 | Helped |
| Add nested tokenization and XL context to Swin-L | Swin-L · iNaturalist 2018 Reptilia | top-1 accuracy: 68.78 -> 73.47 | Helped |
| Add nested tokenization and XL context to Swin-S | Swin-S · xView3-SAR | close-to-shore F1: 46.1 -> 52.8 | Helped |
| Add nested tokenization and XL context to Swin-S | Swin-S · xView3-SAR | aggregate F1: 44.9 -> 55.1 | Helped |
| Add nested tokenization and XL context to Swin-T | Swin-T · xView3-SAR | F1 score: 76.8 -> 81.2 | Helped |
| Add nested tokenization and XL context to Swin-B | Swin-B · xView3-SAR | F1 score: 75.8 -> 77.2 | Helped |
| Use nested tokenization and XL context to process the full image | Swin-B-DetINO · Cityscapes | mAP | Helped |
| Replace ViT context encoder with Hyper (ablation) | Swin-B xT · iNaturalist 2018 Reptilia | top-1 accuracy: 63.26 -> 64.08 | Helped |
| Replace Hyper context encoder with Mamba (ablation) | Swin-B xT · iNaturalist 2018 Reptilia | top-1 accuracy: 64.08 -> 63.73 | Hurt |
Surprising findings
- xT with 256/256 (no extra context) performs worse than the region encoder alone, indicating extra parameters without context can hurt learning.
Limitations
- Throughput drops slightly relative to prior methods except for the Tiny model.
- Mamba cannot be applied in the Transformer-XL recurrent setting and has no Swin-S kernel support.
- Some configurations (e.g., 256/256) add parameters without benefit.
Experimental setup
Datasets: iNaturalist 2018 (Reptilia subset) xView3-SAR Cityscapes ImageNet-1k
Benchmarks: iNaturalist 2018 Reptilia xView3-SAR Cityscapes
Metrics: top-1 accuracy mAP mAP-Large F1 score close-to-shore F1 memory per region throughput
Baselines: Swin-T Swin-S Swin-B Swin-L Hiera-B Hiera-B+ Swin-B-DetINO
Builds on: Transformer-XL Mamba HyperAttention SwinV2 Hiera
Closest papers by meaning
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.87)
- From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection (ICML 2025 · similarity 0.87)
- Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.87)
- XAttention: Block Sparse Attention with Antidiagonal Scoring (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.