Research map / Detection and segmentation / Open-set and few-shot detection
Hand-object and articulated interaction
Model and generate hand-object and articulated object manipulation.
52 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Open-set and few-shot detection, keyword cluster “human · interaction · hoi”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
17 papers in 2023–24 and 10 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.9% to 1.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Visually Indicated Sounds (CVPR 2016 · 373 citations)
- 2018 · Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (ICLR 2018 · 128 citations)
- 2019 · Self-Supervised Learning via Conditional Motion Propagation (CVPR 2019 · 50 citations)
- 2020 · Object Goal Navigation using Goal-Oriented Semantic Exploration (NeurIPS 2020 · 218 citations)
- 2021 · SOON: Scenario Oriented Object Navigation With Graph-Based Exploration (CVPR 2021 · 126 citations)
- 2022 · PONI: Potential Functions for ObjectGoal Navigation With Interaction-Free Learning (CVPR 2022 · 147 citations)
- 2023 · LEGO-Net: Learning Regular Rearrangements of Objects in Rooms (CVPR 2023 · 59 citations)
- 2024 · Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · 31 citations)
- 2025 · ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation (CVPR 2025 · 4 citations)
- 2026 · Towards Autonomous UAV Visual Object Search in City Space: Benchmark and Agentic Methodology (AAAI 2026 · 2 citations)
Earliest papers here
- Visually Indicated Sounds (CVPR 2016 · 373 citations)
- Hierarchical Object Representation for Open-Ended Object Category Learning and Recognition (NeurIPS 2016 · 18 citations)
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (ICLR 2018 · 128 citations)
What has been tried
44 papers here report 471 measured changes (a component added, removed or swapped, compared with a baseline): helped 78%, no clear effect 5%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use 1 prototype per step instead of 2 | EgoPER (all tasks) · EDA | 57.0 -> 55.5 | Hurt | Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · 31 citations) |
| Add LLM general knowledge to episodic observations | Gibson val · SR | 74.4 -> 78.0 | Helped | Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · 22 citations) |
| add a joint-local sensor with radius 0 rather than the full method's setting | GRAB · MPJPE (mm) | 7.24 -> 9.34 | Hurt | GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · 12 citations) |
| Use the PULSE-X latent motion representation as the action space | GRAB-Goal-Test (Cross-Object) · trajectory success | 33.6% -> 94.1% | Helped | Omnigrasp: Grasping Diverse Objects with Simulated Humanoids (NeurIPS 2024 · 8 citations) |
| replace GST-GCN and KIM with Transformer | BEHAVE · FID | 0.095 -> 0.229 | Hurt | ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation (CVPR 2025 · 4 citations) |
| add manipulation feasibility loss | GRAB · plausible rate | 0.82 -> 0.91 | Helped | Physics-Aware Hand-Object Interaction Denoising (CVPR 2024 · 3 citations) |
| use the OMM and PEM rather than removing both modules | KIT MoCap forecasting (0:2) · Overall error (mm) | 52.7 -> 33.7 | Helped | Forecasting Bimanual Object Manipulation Sequences from Unimanual Observations (AAAI 2024 · 2 citations) |
| add the diffusion generation model instead of the generation baseline | OakInk · penetration depth | 0.672 -> 0.554 | Helped | RAGG: Retrieval-Augmented Grasp Generation Model (AAAI 2025 · 1 citation) |
| Use successor features instead of a one-step forward model for the counterfactual interaction reward | Magnetic Blocks · success rate | Helped | Unsupervised Object Interaction Learning with Counterfactual Dynamics Models (AAAI 2024 · 1 citation) | |
| add text tokens | Ice hockey · Mean | 3.35% | Helped | Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction (NeurIPS 2024 · 1 citation) |
| add pose-guided loss | HOIBench · HOI-Score | 34.42 -> 87.39 | Helped | HOComp: Interaction-Aware Human-Object Composition (NeurIPS 2025 · 0 citations) |
| add DINO similarity scores | NYU-v2 · accuracy | 89.52% -> 91.32% | Helped | Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting (NeurIPS 2025 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 27 papers; from OpenAlex and the papers' affiliations).
- Imperial College London · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Shanghai Qi Zhi Institute · 2 papers
- Cornell University · 2 papers
- Carnegie Mellon University · 2 papers
- Meta · 2 papers
- University of California, Berkeley · 2 papers
- ETH Zurich · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Unsupervised object and part discovery (114 papers)
- Human-object interaction detection (66 papers)
- Visual relationship and scene graph detection (63 papers)
Elsewhere in the topic
- Object-level representation learning (219 papers)
- Interactive and referring segmentation (161 papers)
- Open-set and unknown object detection (160 papers)
- Amodal and open-world instance segmentation (137 papers)
- Industrial anomaly detection (117 papers)
- Domain-adaptive object detection (103 papers)
- Open-vocabulary object detection (96 papers)
- Open-vocabulary semantic segmentation (93 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).