Research map / Detection and segmentation / Open-set and few-shot detection
Locality-Aware Zero-Shot Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAIN, which adds a locality adapter and an interaction adapter to a frozen CLIP ViT so HO tokens capture fine-grained human-object interaction cues for zero-shot HOI detection.
Problem
Zero-shot human-object interaction detection adapts CLIP, but CLIP's global image-level representations miss fine-grained object details and human-object interaction patterns, so adapted models can underperform raw CLIP on unseen categories.
Approach
Inserts a Locality Adapter that aggregates neighboring patch tokens with spatial layout priors, and an Interaction Adapter that uses cross-attention between human and object region features to update HO tokens, both injected into each frozen CLIP ViT layer.
Key results
- On HICO-DET under UV setting, LAIN reaches 28.96 unseen mAP versus 26.16 for CLIP and 26.23 for CMMP, outperforming prior zero-shot methods across all five zero-shot settings.
- LAIN with ViT-L (LAIN†) reaches 36.57 RF-UC unseen mAP compared to 28.79 for CLIP ViT-B.
- Ablation on UV setting shows adding both LA and IA raises unseen mAP from 24.88 (baseline) to 30.50 (note: Table 6 full model reports 28.96 unseen).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Locality Adapter to CLIP representations (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 27.71 | Helped |
| Add Interaction Adapter to CLIP representations (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 27.37 | Helped |
| Add both Locality Adapter and Interaction Adapter (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 30.50 | Helped |
| Add visual context to the Locality Adapter (ablation) | LAIN · HICO-DET UV | mAP: 26.77 -> 27.71 | Helped |
| Add spatial layout to the Locality Adapter (ablation) | LAIN · HICO-DET UV | mAP: 26.52 -> 27.71 | Helped |
| Replace Locality Adapter convolutions with Local Attention (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 26.46 | Helped |
| Replace Locality Adapter convolutions with Window Attention (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 26.35 | Helped |
| Add the Interaction Pattern Reasoning Module (ablation) | LAIN · HICO-DET UV | mAP: 24.32 -> 27.37 | Helped |
| Add human/object context extraction (ablation) | LAIN · HICO-DET UV | mAP: 25.64 -> 27.37 | Helped |
| Insert adapters in lower layers 1-6 (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 27.29 | Helped |
| Insert adapters in upper layers 7-12 (ablation) | LAIN · HICO-DET UV | mAP: 24.88 -> 27.83 | Helped |
| Use CLIP ViT-L instead of CLIP ViT-B | LAIN · HICO-DET RF-UC Unseen | mAP: 31.83 -> 36.57 | Helped |
Surprising findings
- Existing CLIP-based zero-shot HOI methods score lower than raw CLIP on unseen classes under the RF-UC and UV settings.
- Inserting adapters into upper ViT layers gives larger gains than lower layers, since upper layers encode global information.
Limitations
- Relies on an off-the-shelf DETR detector for human and object boxes, so detection errors propagate.
- Gains on fully-supervised V-COCO are competitive but not best among zero-shot methods (second-best).
Experimental setup
Datasets: HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP role AP#2
Baselines: CLIP CLIP4HOI HOICLIP ADA-CM CMMP BCOM EoID GEN-VLKT LOGICHOI FCL ATL RLIP
Builds on: CLIP CLIP4HOI HOICLIP DETR ADA-CM
Closest papers by meaning
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.89)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.88)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.88)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.88)
- ContextHOI: Spatial Context Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.88)
- Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.