Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an RL-trained active recognition agent that uses CLIP features, attention-based fusion, and similarity-based policy inputs to perform open-vocabulary object recognition under viewpoint and occlusion variation.
Problem
CLIP's zero-shot recognition is highly sensitive to viewpoint and occlusion, which hurts embodied agents that must classify arbitrary object categories from sequential observations.
Approach
Use PPO to train a movement policy whose inputs are inter-concept similarities to base-class text embeddings and inter-frame similarities, plus a self-attention module that weights frozen CLIP features across frames; the policy and fusion avoid class-specific representations so they generalize to novel classes without fine-tuning CLIP.
Key results
- On ShapeNet with 10 base classes, the agent reaches 53.3% top-1 open-class accuracy versus 29.6% for zero-shot CLIP ViT-B/32.
- On Habitat, the agent reaches 28.0% top-1 open-class accuracy versus 24.9% for CLIP.
- Attention-based fusion outperforms average-feature, average-prediction, max-prediction, and vote fusion (53.3 vs 50.6-51.9 open top-1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the active recognition agent with attention-based evidence fusion instead of passive CLIP | CLIP ViT-B/32 · ShapeNet, 10/45/55 split, open classes | top-1 accuracy: 29.6 -> 53.3 | Helped |
| Use the active recognition agent with attention-based evidence fusion instead of passive CLIP | CLIP ViT-B/32 · Habitat, 10 base/15 novel, open classes | top-1 accuracy: 24.9 -> 28.0 | Helped |
| Use attention-based feature integration instead of average-feature fusion (ablation) | Ours · ShapeNet, 10 base classes, open classes | top-1 accuracy: 50.6 -> 53.3 | Helped |
| Use attention-based feature integration instead of average-prediction fusion (ablation) | Ours · ShapeNet, 10 base classes, open classes | top-1 accuracy: 51.9 -> 53.3 | Helped |
| Use attention-based feature integration instead of max-prediction fusion (ablation) | Ours · ShapeNet, 10 base classes, open classes | top-1 accuracy: 51.6 -> 53.3 | Helped |
| Use attention-based feature integration instead of vote fusion (ablation) | Ours · ShapeNet, 10 base classes, open classes | top-1 accuracy: 51.7 -> 53.3 | Helped |
| Use disentangled policy inputs instead of CLIP features as policy input (ablation) | Ours · ShapeNet, 10 base classes, open classes | top-1 accuracy: 45.4 -> 53.3 | Helped |
Surprising findings
- Random viewpoint sequences do not necessarily improve recognition over single-frame CLIP; a learned intelligent policy is needed.
- Feeding CLIP features into the policy instead of a semantics-disentangled encoder reduces performance, especially on novel classes.
Limitations
- Only viewpoint variation is addressed; lighting and other real-world challenges are left to future work.
- Occlusions used in the investigation are simple random patches relative to real embodied occlusions.
- Inter-concept similarity is a preliminary uncertainty estimate that assumes CLIP rarely makes overconfident misclassifications.
Experimental setup
Datasets: ShapeNetCore HM3D MP3D
Benchmarks: ShapeNetCore Habitat (MP3D test scenes)
Metrics: top-1 accuracy top-3 accuracy
Baselines: CLIP (ViT-B/32) Random policy Largest-step policy Fixation policy Average-feature Average-prediction Max-prediction Vote
Builds on: CLIP PPO active recognition with reinforcement learning
Closest papers by meaning
- I-Perceive: A Foundation Model for Vision-Language Active Perception (NeurIPS 2026 · similarity 0.86)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.86)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.85)
- TANGO: Training-free Embodied AI Agents for Open-world Tasks (CVPR 2025 · similarity 0.85)
- ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning (ICML 2026 · similarity 0.85)
- Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.