Research map / Detection and segmentation / Open-set and few-shot detection
Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InterProDa, which learns Gaussian prompt distributions per HOI category (subject, object, interaction) and samples them into HOI decoder queries, regularized by a dynamic orthogonal loss, to improve HOI detection on HICO-DET and V-COCO.
Problem
Query-transformer HOI detectors struggle to recognize uncommon visual patterns and to distinguish visually similar HOI categories, because their queries have limited capacity to represent intra-category variation and inter-category dependencies.
Approach
Model each HOI category as a learnable Gaussian distribution over multiple soft CLIP prompts, organize the categories into a distribution space with a dynamic orthogonal margin loss, and sample from it with a learnable noise scale to form category distribution queries for the decoder.
Key results
- Achieves 42.67 full mAP and 45.21 rare mAP on HICO-DET (default), and 67.6 scenario 1 AP on V-COCO, reported as SOTA.
- Plugging InterProDa into HOICLIP raises HICO-DET full mAP from 34.69 to 38.04 with about 2M extra parameters.
- Ablation shows the full model improves full mAP from 36.70 (base) to 42.67 on HICO-DET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prompt queries without HOI category priors (ablation) | InterProDa base detector · HICO-DET Default | mAP (Full): 36.70 -> 37.47 | Helped |
| Model prompt collections as Gaussian distributions (ablation) | InterProDa with prompt queries · HICO-DET Default | mAP (Full): 37.47 -> 38.08 | Helped |
| Map prompts to VLM text embeddings (ablation) | InterProDa with distributions · HICO-DET Default | mAP (Full): 38.08 -> 39.56 | Helped |
| Add dynamic orthogonal loss (ablation) | InterProDa with VLM text embeddings · HICO-DET Default | mAP (Full): 39.56 -> 41.85 | Helped |
| Use learnable noise scale in reparameterization (ablation) | InterProDa with orthogonal loss · HICO-DET Default | mAP (Full): 41.85 -> 42.19 | Helped |
| Use HOI-aware prompt structures (ablation) | InterProDa with learnable-noise reparameterization · HICO-DET Default | mAP (Full): 42.19 -> 42.67 | Helped |
| Use dynamic margin Delta instead of no dynamic margin or hard contrastive margins (ablation) | InterProDa orthogonal loss · HICO-DET Default | mAP (Full): 41.62; 41.60 -> 41.85 | Helped |
| Use learnable noise factor rather than VAE reparameterization (ablation) | InterProDa sampling · HICO-DET Default | mAP (Rare): 42.40 -> 45.21 | Helped |
| Replace sampling with repeated distribution mean (ablation) | InterProDa sampling · HICO-DET Default | mAP (Full): 42.67 -> 42.39 | Hurt |
| Use Gaussian rather than naive space basis (ablation) | InterProDa distribution representation · HICO-DET Default | mAP (Full): 40.94 -> 42.01 | Helped |
| Use Fourier rather than Gaussian space basis (ablation) | InterProDa distribution representation · HICO-DET Default | mAP (Full): 42.01 -> 42.40 | Mixed |
| Increase prompts per category K from 2 to 8 (ablation) | InterProDa · HICO-DET Default | mAP (Rare): 43.75 -> 44.77 | Helped |
Surprising findings
- Naive Gaussian-free sampling via max-pooling of prompts underperforms distribution modeling, and a Fourier basis gives similar results to Gaussian bases.
- The Repeat-mu sampling (no noise) variant is competitive on non-rare mAP (42.14), above the full model's 41.92.
Limitations
- Number of prompts K is set to 8 rather than infinity due to memory limits, so the 'infinite' intra-category claim is only theoretical.
- Zero-shot comparisons use the setup of prior work; the paper does not discuss failure cases or compute cost in depth.
Experimental setup
Datasets: HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP scenario 1 AP unseen mAP seen mAP
Baselines: HOTR QPIC CPC GEN-VLKT PViC HOICLIP CQL RLIPv2 LOGICHOI MPHOI RmLR ViPLO
Builds on: CoOp ProDA HOICLIP ADA-CM CLIP
Closest papers by meaning
- QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detection (AAAI 2026 · similarity 0.90)
- Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models (NeurIPS 2023 · similarity 0.90)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.88)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.