Research map / Architectures and efficiency / Attention and transformers

SparseFormer: Sparse Visual Recognition via Limited Latent Tokens

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SparseFormer, a vision transformer that represents images with a small set of latent tokens (down to 49) that adaptively sample image features via bilinear interpolation and RoI adjustment, trained end-to-end with only classification labels.

Problem

Standard vision networks densely process every pixel or patch, incurring computation that scales with input resolution; the paper asks whether recognition can be done sparsely, as human vision appears to do.

Approach

Learnable latent tokens, each with an RoI descriptor, iteratively adjust their RoIs and sample image features at a few bilinear points; a focusing transformer decodes these samples into tokens, and a standard transformer (cortex) operates only on the limited tokens, so cost is nearly independent of input resolution. The model learns where to look from classification loss alone.

Key results

• SparseFormer-T reaches 81.0% ImageNet-1K top-1 with 2.0G FLOPs and 1270 images/s, versus Swin-T at 81.3% with 4.5G FLOPs and 726 images/s.
• Scaling SparseFormer-B with ImageNet-21K pre-training adds 1.0 top-1 point; VideoSparseFormer-S pre-trained on ImageNet-21K reaches 79.8 on Kinetics-400 at 74 GFLOPs.
• Ablation: the adjusting-RoI focusing stages are vital; 'nil' (fixed RoIs) performs markedly worse.

Measured effects reported in the paper

Surprising findings

• Training with raw RGB input instead of early convolution features reached only about 60% top-1, so the early convolution is necessary for stable gradients through bilinear sampling.
• Token-level sampling-density visualizations show the model learns roughly where discriminative foregrounds are from classification labels alone, without localization supervision.

Limitations

• Accuracy is on par with dense models rather than state-of-the-art, and the authors describe the work as an initial step.
• Detection and segmentation results are preliminary and trail DETR and Swin/UperNet baselines.
• Training from raw RGB is unstable, requiring early convolution and careful RoI initialization.

Experimental setup

Datasets: ImageNet-1K ImageNet-21K Kinetics-400 COCO

Benchmarks: ImageNet-1K Kinetics-400 COCO

Metrics: top-1 accuracy throughput FLOPs

Baselines: Swin-T Perceiver ResNet ConvNeXt TimeSFormer ViViT I3D SlowFast

Builds on: Perceiver DETR AdaMixer Swin Transformer dynamic convolution

Closest papers by meaning

• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.89)
• Bootstrapping SparseFormers from Vision Foundation Models (CVPR 2024 · similarity 0.89)
• LookThere! Sparse Vision by Reinforced Selection (NeurIPS 2026 · similarity 0.88)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.87)
• Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering (NeurIPS 2026 · similarity 0.87)
• Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)