Research map / Detection and segmentation / Open-set and few-shot detection
Learn How to See: Collaborative Embodied Learning for Object Detection and Camera Adjusting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a student-teacher framework (STF) where a GPT-style causal transformer student predicts camera adjustment actions and a Siamese teacher gives detection-loss-difference rewards, fine-tuned on a replay buffer of unlabeled trajectories, for embodied object detection.
Problem
Passive object detectors ignore feedback from detection to image acquisition, while active/embodied detection requires expensive interactive data annotation and adapts poorly to new scenarios.
Approach
Treats camera adjustment as sequence prediction with a causally masked GPT-style transformer (instead of RL), uses a teacher network that predicts rewards from the detection-loss difference between consecutive steps, and self-tunes the student on a large unlabeled replay buffer of inference trajectories.
Key results
- STF reaches 88.2 AP on SA and 82.5 on VP, versus 80.9/71.1 for DINO and 72.4/57.9 for FCOS (Table 1).
- STF reaches 85.4 (SA) and 81.3 (VP) by step 5, exceeding the final RL-based performance of 84.7 and 78.2.
- Counting accuracy improves to 95.6% (VP) and 97.2% (SA) versus 90.4% and 93.6% for the best RL-based active method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STF instead of DINO passive detection | Object detection with camera adjustment · Small Airport (SA) | AP: 80.9 -> 88.2 | Helped |
| Use STF instead of DINO passive detection | Object detection with camera adjustment · Virtual Park (VP) | AP: 71.1 -> 82.5 | Helped |
| Use STF instead of the strongest listed active baseline, FSAF+DDQN | Active object detection · Small Airport (SA) | AP: 84.7 -> 88.2 | Helped |
| Use STF instead of the strongest listed active baseline, FSAF+DDQN | Active object detection · Virtual Park (VP) | AP: 78.2 -> 82.5 | Helped |
| Use STF rather than greedy action selection (ablation) | STF · Small Airport (SA) | AP: 78.6 -> 88.2 | Helped |
| Use STF rather than greedy action selection (ablation) | STF · Virtual Park (VP) | AP: 64.4 -> 82.5 | Helped |
| Use detection-loss-based benefit instead of IOU-based benefit (ablation) | STF · Small Airport (SA) | AP: 86.9 -> 87.1 | Helped |
| Use detection-loss-based benefit instead of IOU-based benefit (ablation) | STF · Virtual Park (VP) | AP: 79.3 -> 80.1 | Helped |
| Fine-tune the student network on inference data (ablation) | STF with loss-based benefit · Small Airport (SA) | AP: 87.1 -> 87.8 | Helped |
| Fine-tune the student network on inference data (ablation) | STF with loss-based benefit · Virtual Park (VP) | AP: 80.1 -> 81.9 | Helped |
| Set attention block size to 12 rather than 6 (ablation) | STF · Small Airport (SA) | AP: 88.2 -> 85.4 | Hurt |
| Set attention block size to 12 rather than 6 (ablation) | STF · Virtual Park (VP) | AP: 82.5 -> 79.1 | Hurt |
Surprising findings
- Dense P3 features perform worse than the smaller P6 features (84.9 vs 87.4 on SA), suggesting coarser features suffice for decision-making.
Limitations
- Evaluated only on two simulated aircraft and car datasets from prior work
- Relies on a fixed set of nine discrete camera actions
Experimental setup
Datasets: Small Airport (SA) Virtual Park (VP)
Benchmarks: Small Airport (SA) Virtual Park (VP)
Metrics: AP at IoU 0.50:0.05:0.95 counting accuracy
Baselines: DINO FCOS FSAF FCOS+DDQN FCOS+PPO FCOS+TRPO FSAF+DDQN FSAF+PPO FSAF+TRPO
Builds on: Decision Transformer FCOS FSAF DINO PPO
Closest papers by meaning
- Embodied-DETR: End-to-End Temporal 3D Object Detection in Egocentric Views (ICML 2026 · similarity 0.87)
- Reinforcement Learning for Visual Object Detection (CVPR 2016 · similarity 0.87)
- Revisiting Generative Replay for Class Incremental Object Detection (CVPR 2025 · similarity 0.86)
- Large Self-Supervised Models Bridge the Gap in Domain Adaptive Object Detection (CVPR 2025 · similarity 0.85)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.85)
- Active Object Detection with Knowledge Aggregation and Distillation from Large Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.