Research map / Architectures and efficiency / Attention and transformers

CanViT: Toward Active-Vision Foundation Models

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CanViT, a recurrent active-vision foundation model with a scene-wide canvas updated by asymmetric Canvas Attention, pretrained from scratch via label-free passive-to-active distillation of DINOv3 features on ImageNet-21k glimpse sequences.

Problem

Active computer vision models that process scenes via sequential localized glimpses lag far behind passive vision models, especially on dense prediction, and lack scalable general-purpose architectures and pretraining pipelines.

Approach

A canvas-based dual-stream architecture with scene-relative RoPE and Canvas Attention that is asymmetric (no QKVO projections or MLPs on canvas tokens), trained by distilling dense DINOv3 latents from random glimpse rollouts with policy-agnostic viewpoint sampling.

Key results

• Frozen CanViT-B reaches 38.5% ADE20K mIoU in a single low-resolution glimpse with 20x fewer FLOPs than AME (27.6%), and 45.9% mIoU with additional glimpses.
• CanViT-B achieves 84.5% ImageNet-1k top-1 after fine-tuning, up from the prior active SOTA of 82.2% (AdaptiveNN).
• Pretraining CanViT-B from random initialization takes 166 hours on a single H100 over ~1B glimpses from 13.2M ImageNet-21k scenes.

Surprising findings

• A finer canvas than the glimpse grid improves single-glimpse ADE20K segmentation over the input-matched DINOv3 teacher.
• The F2C policy (worse-than-random ordering) still leaves CanViT's advantage over prior active models intact, suggesting perception rather than viewpoint selection is the bottleneck.

Limitations

• Trained and evaluated only on static scenes.
• Relies on a precomputed passive DINOv3 teacher, which may be costly at very high resolutions or for video.
• Single model size pretrained with modest compute and a dataset over 100x smaller than DINOv3's LVD-1689M.

Experimental setup

Datasets: ImageNet-21k ADE20K ImageNet-1k

Benchmarks: ADE20K ImageNet-1k

Metrics: mIoU top-1 accuracy FLOPs

Baselines: Saccader AME AdaGlimpse AdaptiveNN DINOv3 ViT-B

Builds on: DINOv3 Proteus Perceiver Recurrent Interface Networks LP-FT

Closest papers by meaning

• Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning (ICLR 2025 · similarity 0.88)
• Act2See: Emergent Active Visual Perception for Video Reasoning (CVPR 2026 · similarity 0.87)
• LookThere! Sparse Vision by Reinforced Selection (NeurIPS 2026 · similarity 0.86)
• Seeing More with Less: Human-like Representations in Vision Models (CVPR 2025 · similarity 0.86)
• I-Perceive: A Foundation Model for Vision-Language Active Perception (NeurIPS 2026 · similarity 0.86)
• ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning (ICML 2026 · similarity 0.86)