Research map / Architectures and efficiency / Attention and transformers

Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Learns hierarchical patch-splitting policies that adaptively choose patch sizes per region of an image to reduce ViT token counts, replacing noisy pixel-level heuristics such as local entropy.

Problem

ViT compute is bottlenecked by input token sequence length; fixed patchification wastes tokens on visually simple regions, and prior adaptive patch sizing relies on noisy pixel-level heuristics like local entropy.

Approach

Learns a hierarchical patch-splitting policy that decides, per region, whether to split into finer patches, replacing handcrafted entropy-based decisions with learned ones.

Experimental setup

Baselines: local entropy-based adaptive patch sizing

Builds on: adaptive patch sizing

Closest papers by meaning

• Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.91)
• REOrdering Patches Improves Vision Models (NeurIPS 2025 · similarity 0.89)
• Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.89)
• Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.89)
• An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.88)
• ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.88)