Research map / Vision-language and multimodal / Human and embodied modeling
Differentiable Hierarchical Visual Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a fully differentiable hierarchical superpixel tokenizer for ViTs that selects partitions with information criteria, supports retrofitting pretrained ViTs, and enables raster-to-vector conversion.
Problem
Vision Transformers use fixed square patch tokens that ignore image structure, and existing subobject/superpixel tokenizers are not end-to-end differentiable or not backward compatible with standard ViTs.
Approach
Build a hierarchy of pixel-level superpixels by iterative kernel-based vertex merging, select the optimal partition via information criteria with a graph-volume-based degrees-of-freedom estimate, and extract differentiable token features via a mean-injection trick with learnable foreground/background blending so standard ViTs can be used unchanged.
Key results
- Retrofitted DEiT3-RF ViT-B16 reaches 83.9% Acc@1 on ImageNet-Val vs 82.6% for the DEiT3 baseline.
- Trained from scratch, the proposed ViT-B16 reaches 83.2% Acc@1 vs 82.6% for DEiT3 patch tokens, while SPiT reaches 80.4% (base).
- Ablation: removing the CNN feature projection drops DEiT3-RF ViT-S16 accuracy from 80.1% to 76.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · ImageNet1k val | Acc@1: 82.6 -> 83.9 | Helped |
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · ImageNet1k ReaL | Acc@1: 87.7 -> 88.3 | Helped |
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · ImageNet-v2 | Acc@1: 72.6 -> 73.2 | Helped |
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · Caltech | Acc@1: 90.3 -> 91.2 | Helped |
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · CUB | Acc@1: 72.1 -> 74.4 | Helped |
| Replace patch tokens with differentiable hierarchical superpixel tokens | Retrofitted ViT-B16 · Cars | Acc@1: 35.1 -> 34.7 | Hurt |
| Replace patch tokens with differentiable hierarchical superpixel tokens | ViT-B16 trained from scratch · ImageNet1k | Acc@1: 82.6 -> 83.2 | Helped |
| Replace the CNN encoder with a linear projection (ablation) | Differentiable hierarchical tokenization · ImageNet1k, retrofitted ViT-S16 | Acc@1: 80.1 -> 76.9 | Hurt |
| Remove mask blending (ablation) | Differentiable hierarchical tokenization · ImageNet1k, retrofitted ViT-S16 | Acc@1: 80.1 -> 71.8 | Hurt |
| Remove mask blending (ablation) | Differentiable hierarchical tokenization · ImageNet1k, from-scratch ViT-S16 | Acc@1: 80.0 -> 78.9 | Hurt |
| Remove beta-mask features (ablation) | Differentiable hierarchical tokenization · ImageNet1k, retrofitted ViT-S16 | Acc@1: 80.1 -> 79.7 | Hurt |
| Remove kernel aggregation (ablation) | Differentiable hierarchical tokenization · ImageNet1k, retrofitted ViT-S16 | Acc@1: 80.1 -> 80.0 | No clear effect |
Surprising findings
- Removing IC-based pruning increases token count without improving accuracy, and in the from-scratch setting it slightly lowers accuracy (79.9 vs 80.0).
Limitations
- Lower throughput than canonical ViTs due to superpixel computation overhead.
- Gaussian pixel assumption is approximate; residuals are heavier-tailed.
- Weaker at very low resolutions such as CIFAR 32x32, where it can behave like square patches.
Experimental setup
Datasets: ImageNet1k COCO-Stuff
Benchmarks: ImageNet-Val ImageNet-ReaL ImageNet-v2 Caltech256 CUB200 Stanford Cars ADE20k COCO-Stuff ImageNet reconstruction MSE/SSIM
Metrics: Acc@1 kNN accuracy mIoU MSE SSIM
Baselines: DEiT3 TIMM ViT-B32 SPFormer SPiT SuiT DEiT
Builds on: DEiT3 SPiT SuiT TokenCut
Closest papers by meaning
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.87)
- Subobject-level Image Tokenization (ICML 2025 · similarity 0.86)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.86)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.86)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.