Research map / 3D vision / 3D generation
Learning to Select Views for Efficient Multi-View Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVSelect, a light Q-learning view-selection module that sequentially picks the next-best camera view for multi-view classification and detection, using only 2-3 of N views with near full-view accuracy.
Problem
Multi-view perception is computationally expensive on resource-limited devices because all N camera views must be processed; the task is to select a small, most helpful subset of views efficiently.
Approach
Formulate sequential next-best-view selection as an RL problem (Q-learning with a light MVSelect controller that conditions on chosen-camera one-hot encodings and max-pooled observed features), so non-differentiable view selection can be trained via trial and error, optionally jointly with the task network, and pipelined for GPU parallelism.
Key results
- On ModelNet40 (12 views), joint-trained MVSelect with T=2 views reaches 94.3% accuracy versus 94.5% using all 12 views.
- On Wildtrack, MVSelect with fixed task network reaches 80.0 MODA at T=3; joint training reaches 88.6 MODA versus 90.0 with all 7 views.
- Efficiency: T=2 of 12 views reduces ModelNet40 FLOPs from 21.9G to 3.6G and raises throughput from 196.2 to 507.9 instances/s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train MVSelect and MVCNN rather than keeping the task network fixed | MVCNN† · ModelNet40 (12 views) | accuracy: 88.2 -> 94.3 | Helped |
| Jointly train MVSelect and MVCNN rather than keeping the task network fixed | MVCNN† · ModelNet40 (20 views) | accuracy: 79.6 -> 94.4 | Helped |
| Jointly train MVSelect and MVCNN rather than keeping the task network fixed | MVCNN† · ScanObjectNN (12 views) | accuracy: 80.0 -> 84.1 | Helped |
| Jointly train MVSelect and MVDet rather than keeping the task network fixed | MVDet† · Wildtrack | MODA: 80.0 -> 88.6 | Helped |
| Jointly train MVSelect and MVDet rather than keeping the task network fixed | MVDet† · MultiviewX | MODA: 78.7 -> 88.1 | Helped |
| Use 2 selected views instead of all views with the fixed task network | MVCNN† · ModelNet40 (12 views) | accuracy: 94.5 -> 88.2 | Hurt |
| Use 2 selected views instead of all views with the fixed task network | MVCNN† · ModelNet40 (20 views) | accuracy: 96.5 -> 79.6 | Hurt |
| Use 2 selected views instead of all views with the fixed task network | MVCNN† · ScanObjectNN (12 views) | accuracy: 86.1 -> 80.0 | Hurt |
| Use 3 selected views instead of all views with the fixed task network | MVDet† · Wildtrack | MODA: 90.0 -> 80.0 | Hurt |
| Use 3 selected views instead of all views with the fixed task network | MVDet† · MultiviewX | MODA: 93.0 -> 78.7 | Hurt |
| Add a camera branch to MVSelect (ablation) | MVCNN† · ModelNet40 (12 views) | accuracy: 88.2 -> 88.2 | No clear effect |
| Add a camera branch to MVSelect (ablation) | MVDet† · Wildtrack | MODA: 79.1 -> 80.0 | Helped |
Surprising findings
- For multi-view detection, the learned MVSelect policy tends to pick the same camera for a given initial view and does not outperform the dataset-level oracle, which uses the test set.
- Transformer-based MVSelect with more parameters did not improve performance over the lightweight design.
Limitations
- Detection performance with fixed task network lags behind joint training and dataset-level oracle; policy ignores uncovered areas outside FoV.
- Assumes known, fixed camera layout; moving-camera setups are not handled (random object pose only partially addressed).
Experimental setup
Datasets: ModelNet40 ScanObjectNN Wildtrack MultiviewX
Benchmarks: ModelNet40 ScanObjectNN Wildtrack MultiviewX
Metrics: instance-averaged accuracy MODA
Baselines: MVCNN GVCNN MHBN RotationNet RelationNet ViewGCN MVTN SEE++ MVDet SHOT MVDeTr random selection
Builds on: MVCNN MVDet DQN temporal difference learning
Closest papers by meaning
- LookThere! Sparse Vision by Reinforced Selection (NeurIPS 2026 · similarity 0.85)
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.84)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.84)
- CoRDS: Coreset-Based Representative and Diverse Selection for Streaming Video Understanding (NeurIPS 2026 · similarity 0.83)
- MiCo: Multi-image Contrast for Reinforcement Visual Reasoning (NeurIPS 2025 · similarity 0.83)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.