Research map / Video / Video-language understanding
Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns to select the most informative camera view per clip in multi-view instructional videos without best-view labels, using view-dependent captioning accuracy against narrations as weak supervision plus an auxiliary relative camera pose loss.
Problem
Selecting the most informative viewpoint at each timestep of a multi-view instructional video normally requires expensive best-view labels or hand-coded heuristics; this work seeks to learn view selection without such labels.
Approach
Use the relative accuracy of view-specific captioner outputs compared against view-agnostic ground-truth narrations as pseudo-labels for the best view, aggregated across multiple captioners, and train a TimeSformer-based view selector with an auxiliary relative camera pose prediction loss to increase view sensitivity.
Key results
- On Ego-Exo4D, LANGVIEW reaches 13.5 CIDEr and 33.7 V-IoU versus 12.9 for the best heuristic baseline (Body-area) on CIDEr; on LEMMA it reaches 42.7 CIDEr vs 42.1 for Body-area.
- In human pairwise evaluation, the selected views beat Body-area on Ego-Exo4D with 55.1% win vs 39.3% loss.
- Ablation on Ego-Exo4D shows removing the pose predictor lowers N-IoU from 39.2 to 38.7 and removing the rank aggregator lowers N-IoU to 38.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LANGVIEW instead of Body-area view selection | Ego-Exo4D · Ego-Exo4D | CIDEr: 12.9 -> 13.5 | Helped |
| Use LANGVIEW instead of Body-area view selection | Ego-Exo4D · Ego-Exo4D | METEOR: 48.2 -> 48.4 | Helped |
| Use LANGVIEW instead of Hand-object view selection | Ego-Exo4D · Ego-Exo4D | V-IoU: 33.6 -> 33.7 | Helped |
| Use LANGVIEW instead of Body-area view selection | Ego-Exo4D · Ego-Exo4D | N-IoU: 37.2 -> 39.2 | Helped |
| Use LANGVIEW instead of Body-area view selection | Ego-Exo4D · Ego-Exo4D | NC-IoU: 31.1 -> 32.9 | Helped |
| Use LANGVIEW instead of Body-area view selection | LEMMA · LEMMA | CIDEr: 42.1 -> 42.7 | Helped |
| Use LANGVIEW instead of Body-area view selection | LEMMA · LEMMA | METEOR: 73.8 -> 74.4 | Helped |
| Use LANGVIEW instead of Body-area view selection | LEMMA · LEMMA | V-IoU: 38.6 -> 40.1 | Helped |
| Use LANGVIEW instead of Hand-object view selection | LEMMA · LEMMA | N-IoU: 41.5 -> 42.9 | Helped |
| Use LANGVIEW instead of Hand-object view selection | LEMMA · LEMMA | NC-IoU: 17.9 -> 18.9 | Helped |
| Add relative camera pose prediction (ablation) | LANGVIEW · Ego-Exo4D | CIDEr: 13.2 -> 13.5 | Helped |
| Use the proposed view classification loss instead of the ablated loss (ablation) | LANGVIEW · Ego-Exo4D | N-IoU: 37.4 -> 39.2 | Helped |
Surprising findings
- Longest-caption baseline, which uses the captioner's output length as an informativeness proxy, generally underperforms naive baselines.
- Random-exo improves over Ego-only on Ego-Exo4D but performs much worse on LEMMA.
Limitations
- Selector sometimes fails when multiple views are of similar high quality.
- Pseudo-labels depend on off-the-shelf captioners that can hallucinate and on CIDEr-based scoring.
- Evaluated only on two datasets with narrations; human evaluation used 10 participants and 70 pairs per study.
Experimental setup
Datasets: Ego-Exo4D LEMMA
Benchmarks: Ego-Exo4D LEMMA
Metrics: CIDEr METEOR V-IoU N-IoU NC-IoU human pairwise win/loss/tie
Baselines: Ego-only Random Random-exo Hand-object Body-area Joint-count Snap angles Longest-caption
Builds on: VideoLLaMA VideoChat2 TimeSformer VLocNet++
Closest papers by meaning
- MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.84)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.84)
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.84)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.83)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.