Research map / Detection and segmentation / Open-set and few-shot detection
Subobject-level Image Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces subobject-level adaptive image tokenization and EPOC, a 3.7M-parameter boundary-detection plus watershed tokenizer, showing better morphology alignment, monosemanticity, efficiency, and faster VLM convergence than patch and object-level tokenizers.
Problem
Patch-based image tokenization is non-adaptive, causing polysemantic tokens in large patches or redundant tokens in small patches, which hurts VLM learning efficiency and generalization.
Approach
Treat subobject segmentation as pixel-wise boundary detection with a tiny SegFormer-b0 and convert boundaries to a complete panoptic partition via a revisited watershed algorithm whose seed threshold controls granularity, giving O(1) cost in the number of tokens.
Key results
- EPOC uses 3.7M parameters versus SAM ViT-H's 641M and reaches up to 17.1 FPS with under 10% GPU utilization on a V100 with 30 CPU cores.
- Subobject-level tokenizers (EPOC, SLIC) reach monosemanticity above 90% in intrinsic evaluation, while object-level panoptic models remain mostly below 60%.
- Subobject tokenization converges faster and reaches lower validation perplexity with far fewer visual tokens than patch tokenization across datasets; patch-based VLMs fail to converge with VAE embeddings on CLEVR-cap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EPOC (t=0.5) instead of SAM ViT-H | Image tokenizer inference · Tokenizer inference | Maximum FPS: 0.4 -> 17.1 | Helped |
| Increase EPOC watershed threshold from t=0.1 to t=0.5 (ablation) | EPOC · Tokenizer inference | Maximum FPS: 13.4 -> 17.1 | Helped |
| Set EPOC watershed threshold to t=0.3 instead of t=0.1 (ablation) | EPOC · Tokenizer inference | Maximum FPS: 13.4 -> 16.2 | Helped |
| Use subobject-level tokenization rather than patch tokenization | Vision-language models · ImageNet-1K, ShareGPT-4V, Pixmo-cap, and CLEVR-cap | Validation perplexity | Helped |
| Use EPOC-based tokenization rather than random patch dropout | Vision-language models · Token truncation evaluation | Validation perplexity | Helped |
| Use subobject tokenization rather than patch tokenization with VAE features | Vision-language models · CLEVR-cap | Convergence | Helped |
| Use EPOC instead of panoptic segmentation or SAM ViT-B for zero-shot segmentation | Image tokenizer · PPP and PIN++ | Boundary precision and recall | Helped |
Surprising findings
- Patch-based VLMs failed to converge with weaker VAE features even with a 1.7B LLM, while adaptive tokenizers trained normally.
- CLIP underperformed DINOv2 as a token embedding in the CLEVR-cap setup.
Limitations
- SegFormer-b0 outputs at quarter resolution, so very fine structures such as hair or small text may not be segmented precisely.
- Extrinsic VLM experiments use LLMs up to 1.7B parameters, far below the largest LLMs.
- The separate segmentation process adds overhead that matters for latency-sensitive applications; adaptive tokenization also discards intra-token spatial structure.
Experimental setup
Datasets: SA-1B ImageNet-1K ShareGPT-4V Pixmo-cap CLEVR-cap COCO ADE-20K Pascal Panoptic Parts PartImageNet++ CLEVR
Benchmarks: COCONut ADE-20K Pascal Panoptic Parts PartImageNet++ SA-1B ImageNet-1K ShareGPT-4V Pixmo-cap CLEVR-cap
Metrics: boundary precision boundary recall token monosemanticity score FPS GPU utilization validation perplexity average training loss
Baselines: patch tokenization Mask2Former OneFormer SLIC superpixel SAM ViT-B SAM ViT-H FastSAM MobileSAMv2
Builds on: SAM SegFormer Mask2Former watershed algorithm DINOv2
Closest papers by meaning
- Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2026 · similarity 0.88)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.87)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.87)
- Single-pass Adaptive Image Tokenization for Minimum Program Search (NeurIPS 2025 · similarity 0.87)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.