Research map / Vision-language and multimodal / Multimodal LLMs

What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ImagineAgent, a Qwen3-VL-8B agent trained with CoT SFT on hicodet-6K and tool-augmented GRPO (crop, RAG explanations, BAGEL outpainting/viewpoint transform) with a composite F1-based reward for open-vocabulary HOI detection.

Problem

Open-vocabulary human-object interaction detection with MLLMs suffers from perceptual-cognitive gaps, cross-modal hallucination from textual priors, and occlusion-induced ambiguity from limited viewpoints.

Approach

An agentic MLLM that performs cognitive mapping over all human-object pairs, dynamically invokes multimodal tools (image crop, retrieval-based action/scene explanation, and generative outpainting/viewpoint transform via BAGEL), and is optimized with GRPO using a composite reward combining Hungarian-matched HOI F1, format, and tool-usage terms.

Key results

• Achieves 36.76% Full mAP on HICO-DET under simulated zero-shot, a 13.64-point gain over INP-CC as reported by the authors.
• Achieves 20.84% Full mAP on SWIG-HOI, 4.10 points above INP-CC.
• SFT alone reaches 27.23% mAP on HICO-DET; SFT plus RL reaches 36.76%.

Surprising findings

• GPT-5 reaches only 23.18% Full mAP on HICO-DET, below the specialized agent built on an 8B model.
• Direct RL without SFT led to declining tool invocation frequency during rollouts, motivating the SFT cold start.

Limitations

• Generative tools (BAGEL) can introduce hallucinated visual content, with about 88% semantic fidelity on sampled generated views.
• Remaining errors are dominated by incomplete HOI matching (39.7%) and object classification errors.

Experimental setup

Datasets: HICO-DET SWiG-HOI

Benchmarks: HICO-DET SWIG-HOI

Metrics: mAP

Baselines: INP-CC HOICLIP CLIP4HOI CMD-SE Qwen2.5-VL-7B GPT-4o Gemini 3.0 Pro GPT-5

Builds on: GRPO DeepSeek-R1 INP-CC

Closest papers by meaning

• Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.88)
• ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning (CVPR 2026 · similarity 0.87)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.87)
• V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning (ICML 2026 · similarity 0.87)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)