Research map / Detection and segmentation / Open-set and few-shot detection
Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BCOM, a two-stage HOI detector that combines CLIP semantic adapters with spatial adapters on a frozen DETR backbone, plus occluded part extrapolation and conditional contextual mining, reaching new SOTA on HICO-DET.
Problem
Recognizing subtle and intricate human-object interactions is hard because CLIP-based representations capture high-level semantics but miss fine-grained spatial detail, and occlusion in crowded scenes further hinders interaction recognition.
Approach
Complements frozen CLIP semantic adapters with multi-scale spatial adapters attached to a frozen DETR backbone (bilateral adaptation), adds a feature-level occlusion-and-reconstruction objective (OPE), and uses a cross-attention module (CCM) conditioned on the paired instance to mine contextual spatial clues.
Key results
- BCOM reaches 39.34 full mAP on HICO-DET default setting, +0.94 over prior SOTA ADA-CM (38.40).
- On HICO-DET zero-shot Rare-First, BCOM gets 33.74 full mAP vs 33.01 for ADA-CM.
- Ablation: removing spatial adaptation drops HICO-DET full mAP by 6.32 (to 33.02 from 39.34 implied); late fusion beats early fusion (39.34 vs 36.90).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial adaptation to semantic adaptation (ablation) | BCOM · HICO-DET | mAP: 36.33 -> 39.34 | Helped |
| Add semantic adaptation to spatial adaptation (ablation) | BCOM · HICO-DET | mAP: 33.02 -> 39.34 | Helped |
| Use late rather than early fusion of semantic and spatial features (ablation) | BCOM · HICO-DET | mAP: 36.90 -> 39.34 | Helped |
| Add OPE (ablation) | BCOM · HICO-DET, more than 10 instances/image | mAP: 16.88 -> 18.53 | Helped |
| Use 2 OPE layers rather than none (ablation) | BCOM · HICO-DET | mAP: 38.57 -> 39.34 | Helped |
| Increase OPE depth from 2 to 3 layers (ablation) | BCOM · HICO-DET | mAP: 39.34 -> 39.00 | Hurt |
| Use pixel-level rather than patch-level occlusion (ablation) | BCOM OPE · HICO-DET | mAP: 38.99 -> 39.34 | Helped |
| Add CCM (ablation) | BCOM · HICO-DET | mAP: 37.44 -> 39.34 | Helped |
| Increase CCM depth from 2 to 3 layers (ablation) | BCOM · HICO-DET | mAP: 39.34 -> 39.18 | Hurt |
| Use BCOM rather than ADA-CM | HOI detection · HICO-DET Default, Full | mAP×100: 38.40 -> 39.34 | Helped |
| Use BCOM rather than MUREN | HOI detection · V-COCO Scenario 1 | mAP×100: 68.8 -> 65.8 | Hurt |
| Use BCOM rather than ADA-CM | HOI detection · HICO-DET zero-shot RF, Full | mAP: 33.01 -> 33.74 | Helped |
Surprising findings
- Spatial adaptation alone performs worse than semantic adaptation alone (33.02 vs 36.33 full mAP), yet combining them gives the largest gain.
- Patch-level masking used in MAE/BEiT is less effective than pixel-level masking on feature maps (38.99 vs 39.34).
Limitations
- On V-COCO the method is second-best, behind MUREN, which has a heavier architecture and longer schedule.
- CCM with 3 layers improves rare-class performance but lowers overall mAP, suggesting overfitting risk.
- Occlusion is simulated at feature level via random masking rather than from real occluded instances.
Experimental setup
Datasets: HICO-DET V-COCO MS-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP
Baselines: ADA-CM ViPLO GEN-VLKT HOI-CLIP PViC AGER MUREN SDT CDN UPT
Builds on: ADA-CM UPT DETR CLIP MAE
Closest papers by meaning
- ContextHOI: Spatial Context Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.91)
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.89)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.88)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.88)
- HOIMamba: Efficient Mamba-based Disentangled Progressive Learning for HOI Detection (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.