Research map / Vision-language and multimodal / Multimodal LLMs
Test-Time Canonicalization by Foundation Models for Robust Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FoCal, a training-free test-time method that generates transformed variants of an input and selects the most visually typical one by minimizing a CLIP plus Stable Diffusion energy, improving robustness of CLIP, SAM and others to rotations, lighting, and viewpoint changes.
Problem
Vision foundation models and task networks are brittle under transformations such as 2D/3D rotations, color and contrast shifts, and day-night changes, and existing fixes require transform-specific training or architectures.
Approach
Treat invariance as test-time canonicalization: vary the input over a transformation set (via generators like TRELLIS or direct 2D transforms), then rank candidates by a combined energy from CLIP classifier logits and diffusion denoising error (SD-2-base), passing the minimum-energy view to the downstream model; continuous spaces are searched with Bayesian optimization.
Key results
- On Objaverse-LVIS, FoCal raises worst-view accuracy from 12.0% to 62.0% over OV-Seg.
- On CIFAR10/100/STL10 rotated inputs, FoCal matches or beats PRLC-trained canonicalizers in every setting zero-shot (e.g., ViT CIFAR100 C8 rotated 84.4 vs 82.2 for PRLC).
- On ImageNet with ViT, FoCal improves rotated accuracy by +11.4 points over PRLC (71.2 vs 60.5 in table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Canonicalize views with FoCal rather than use TRELLIS | OV-Seg on CO3D · CO3D threshold 0.3 | accuracy: 45.9% -> 49.5% (+3.6%) | Helped |
| Canonicalize views with FoCal rather than use TRELLIS | OV-Seg on CO3D · CO3D threshold 0.5 | accuracy: 53.4% -> 55.3% (+1.9%) | Helped |
| Average logits over the top 10 ranked viewpoints instead of 10 random views | OV-Seg on Objaverse-LVIS · Objaverse-LVIS quality-filtered | mean accuracy: 76.4% -> 84.5% | Helped |
| Use FoCal canonicalization instead of PRLC | ResNet50 (PRLC-Trained) · CIFAR10, C8 rotations | accuracy: 95.1 -> 95.6 (+0.5%) | Helped |
| Use FoCal canonicalization instead of PRLC | ResNet50 (PRLC-Trained) · CIFAR100, C8 rotations | accuracy: 81.8 -> 82.2 (+0.4%) | Helped |
| Use FoCal canonicalization instead of PRLC | ResNet50 (PRLC-Trained) · STL10, C8 rotations | accuracy: 94.1 -> 95.6 (+0.6%) | Helped |
| Use FoCal rather than PRLC canonicalization | SAM on COCO · COCO, C4 rotations | pose accuracy: 86.8 -> 88.9 (+2.1%) | Helped |
| Use FoCal rather than PRLC canonicalization | ViT (Vanilla-Trained) · ImageNet, C8 rotations | accuracy: 60.5 -> 71.2 (+11.4) | Helped |
| Use FoCal rather than PRLC canonicalization | CLIP (Vanilla-Trained) · ImageNet, C8 rotations | accuracy: 69.6 -> 73.9 (+4.3) | Helped |
| Add diffusion energy to CLIP energy (ablation) | FoCal pose estimation · 2D rotation | pose error: 37.1° -> 13.5° | Helped |
| Add CLIP energy to diffusion energy (ablation) | FoCal pose estimation · 2D rotation | pose error: 22.6° -> 13.5° | Helped |
| Replace canonicalization with test-time augmentation (ablation) | CLIP · CIFAR10 | accuracy: 93.7 -> 82.8 (-10.9) | Hurt |
Surprising findings
- TRELLIS and random 2D rotations perform surprisingly well as canonicalizers on CO3D, and FoCal's gain over TRELLIS is modest (+3.6% and +1.9%).
- On color correction (RCC), FoCal is far behind specialized methods (6.4° vs 1.3° median angular error), since typicality does not equal color neutrality.
Limitations
- Computational cost scales with number of candidate transforms and diffusion steps (about 56x FLOPs vs naive for 2D rotation).
- Assumes transformations are invertible for formal invariance; non-invertible transforms like viewpoint shifts give only approximate invariance.
- Requires the user to select which transformation generator(s) to use.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 ImageNet COCO Objaverse-LVIS CO3D KITTI RCC
Benchmarks: CIFAR10 CIFAR100 STL10 ImageNet COCO Objaverse-LVIS CO3D RCC
Metrics: accuracy pose accuracy pose error mAP
Baselines: PRLC Rotation augmentation Test-time augmentation OV-Seg TRELLIS
Builds on: learned canonicalization (LC) PRLC energy-based models TRELLIS
Closest papers by meaning
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.86)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.86)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.85)
- Unleashing Perception-Time Scaling to Multimodal Reasoning Models (ICLR 2026 · similarity 0.85)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.