Research map / Architectures and efficiency / Attention and transformers

Learning Dynamic Evidence Routes for Vision Transformer Probing

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SSMProbe, a Sinkhorn-learned token routing followed by a diagonal S4 decoder on frozen ViT patch tokens, used to audit the geometry of the learned evidence route across MAE, BEiT, DINOv2, and supervised ViT.

Problem

Frozen ViT probes usually use permutation-invariant pooling (GAP or CLS) or opaque content-dependent readouts (self-attention), which hide where evidence is extracted from and how it is organized before aggregation.

Approach

Replace invariant pooling with a Sinkhorn-learned soft permutation that routes patch tokens into a diagonal S4 LTI decoder, whose fixed position-dependent coefficients make the induced evidence route inspectable; route geometry is measured with a held-out spatial-predictability diagnostic (ERSI) against an exchangeable location-bias null.

Key results

• On frozen MAE ImageNet-1K, Sinkhorn-learned route reaches 70.33% top-1 vs. GAP 58.10% and CLS 56.65%, with random-like route geometry (R_k about 0.99-1.01).
• On CUB, ERSI_20 is 0.000 for MAE versus 0.175 for BEiT, 0.113 for ViT, and 0.484 for DINOv2, indicating dispersed MAE routes and spatially organized routes for the others.
• DINOv2 CLS reaches 89.49% on CUB, the strongest readout, while Sinkhorn reaches 82.28%; fixed scans underperform random-fixed control.

Surprising findings

• On MAE CUB, the Sinkhorn route (56.67%) and Transformer readouts are far above fixed scans, yet the route geometry is indistinguishable from random (ERSI 0.000).
• On ImageNet-1K MAE, a random fixed permutation with S4 (63.98%) matches raster scan (63.94%), suggesting spatial scan order gives no advantage.

Limitations

• Learned Sinkhorn routes reflect readout-induced utility rather than intrinsic token ordering; not a universal ranking.
• Route geometry is not robust to input transformations such as center crop and Gaussian noise on CUB.
• Accuracy of SSMProbe is below a 3x wider Transformer reference on ImageNet-1K MAE; it is a diagnostic not a SOTA probe.

Experimental setup

Datasets: ImageNet-1K CUB-200-2011 Stanford Cars

Benchmarks: ImageNet-1K CUB-200-2011 Stanford Cars

Metrics: top-1 accuracy ERSI (bits per routed patch) compactness ratio R_k fitted spatial scale sigma

Baselines: GAP CLS Attention Pool Content-Weighted Pool Top-k Pool Transformer DeepSets Bi-GRU Random-Fixed Perm + S4 Random-Dynamic Perm + S4 Raster scan 4-dir VMamba

Builds on: S4 Sinkhorn-Knopp NeuralSort SoftSort VMamba SS2D

Closest papers by meaning

• Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces (NeurIPS 2026 · similarity 0.85)
• Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.85)
• Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.84)
• Statistical Test for Attention Maps in Vision Transformers (ICML 2024 · similarity 0.84)
• VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning (ICML 2026 · similarity 0.84)
• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.84)