Research map / Detection and segmentation / Open-set and few-shot detection
Active Object Detection with Knowledge Aggregation and Distillation from Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KAD, which aggregates LLM-generated interaction descriptions, diffusion-generated images, and ground-truth boxes into an oracle query for a teacher decoder, then distills attention and embeddings into a query-only student detector for active object detection.
Problem
Active object detection (locating objects undergoing state changes) is hard because visual changes are subtle, distractors of the same category are present, and intra-class appearance varies greatly; at inference the active object category is unknown.
Approach
Build a teacher decoder with an oracle query that fuses semantic interaction priors (from GPT descriptions), fine-grained visual priors (from diffusion-generated images), and spatial priors (ground-truth box), then distill its decoder outputs and cross-attention into a student that uses learnable queries only at inference.
Key results
- On Ego4D with Swin-L backbone, KAD reaches 40.5 AP vs. 36.4 for InternVideo (Swin-L).
- On Epic-Kitchens with Swin-L, KAD reaches 35.2 AP vs. 28.3 for InternVideo.
- On 100DOH, KAD reaches 31.2 AP75 vs. 29.9 for Seq-Voting, using only spatial priors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KAD instead of InternVideo with Swin-L | Active object detection · Ego4D Val-Set | AP: 36.4 -> 40.5 | Helped |
| use KAD instead of InternVideo with Swin-L | Active object detection · Epic-Kitchens Val-Set | AP: 28.3 -> 35.2 | Helped |
| add spatial priors (ablation) | VBD on Ego4D · Ego4D | AP: 35.9 -> 36.1 | Helped |
| add semantic priors (ablation) | VBD on Ego4D · Ego4D | AP: 35.9 -> 36.5 | Helped |
| add visual priors (ablation) | VBD on Ego4D · Ego4D | AP: 35.9 -> 36.0 | No clear effect |
| add spatial, semantic, and visual priors (ablation) | VBD on Ego4D · Ego4D | AP: 35.9 -> 40.5 | Helped |
| add embedding distillation (ablation) | VBD on Ego4D · Ego4D | AP: 35.9 -> 38.3 | Helped |
| add attention distillation to embedding distillation (ablation) | VBD on Ego4D · Ego4D | AP: 38.3 -> 40.5 | Helped |
| use 10 generated interaction descriptions instead of none (ablation) | KAD on Ego4D · Ego4D | AP: 37.3 -> 40.5 | Helped |
| use 100 generated images instead of none (ablation) | KAD on Ego4D · Ego4D | AP: 37.9 -> 40.5 | Helped |
| use attentive aggregation instead of max-pooling (ablation) | KAD on Ego4D · Ego4D | AP: 39.2 -> 40.5 | Helped |
Surprising findings
- Spatial-only oracle query (ground-truth box) gives only a small AP gain (+0.2) on Ego4D, while adding semantic and visual priors contributes most of the improvement.
Limitations
- Semantic and visual priors cannot be built for datasets lacking object categories (MECCANO, 100DOH), so only spatial priors are used there.
- Oracle query relies on ground-truth active object category and box during training, and the teacher needs GPT and diffusion-generated priors.
Experimental setup
Datasets: Ego4D Epic-Kitchens MECCANO 100DOH
Benchmarks: Ego4D Epic-Kitchens MECCANO 100DOH
Metrics: AP AP50 AP75 AP25
Baselines: CenterNet Faster R-CNN 100DOH-model DETR InternVideo Seq-Voting PPDM HOTR
Builds on: DETR DINO InternVideo CLIP Seq-Voting
Closest papers by meaning
- Distillation Using Oracle Queries for Transformer-Based Human-Object Interaction Detection (CVPR 2022 · similarity 0.87)
- Ins-DetCLIP: Aligning Detection Model to Follow Human-Language Instruction (ICLR 2024 · similarity 0.86)
- Learn How to See: Collaborative Embodied Learning for Object Detection and Camera Adjusting (AAAI 2024 · similarity 0.85)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.85)
- VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion (NeurIPS 2025 · similarity 0.84)
- InstructDET: Diversifying Referring Object Detection with Generalized Instructions (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.