Research map / Detection and segmentation / Open-set and few-shot detection
Predicate Hierarchies Improve Few-Shot State Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PHIER, a state classification model that learns LLM-inferred predicate hierarchies in a hyperbolic image-predicate latent space to enable few-shot generalization to novel object-predicate pairs and predicates.
Problem
Few-shot, out-of-distribution state classification of objects and relations (novel predicates and unseen object-predicate combinations) for robot planning, where the combinatorial space of object-predicate queries makes exhaustive training data infeasible.
Approach
Encode predicate hierarchies in a joint image-predicate space using an object-centric MaskCLIP-based encoder, LLM-derived predicate triplet and norm regularization losses, and a Poincaré-ball hyperbolic encoder and distance metric so that specificity maps to norm and similarity to distance.
Key results
- On CALVIN OOD, PHIER reaches 0.899 accuracy vs 0.674 for the best prior trained method (Re-Attention), a 22.5 point gain.
- On BEHAVIOR OOD, PHIER reaches 0.820 vs 0.712 for BUTD, an 8.3 point gain.
- Zero-shot real-world transfer on BEHAVIOR Vision Suite: PHIER 0.608 vs GPT-4V 0.712 (GPT-4V higher by 10.4 points); few-shot PHIER 0.703.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PHIER instead of Re-Attention | few-shot state classification · CALVIN OOD | accuracy: 0.674 -> 0.899 | Helped |
| Use PHIER instead of RelViT | few-shot state classification · BEHAVIOR OOD | accuracy: 0.737 -> 0.820 | Helped |
| Use PHIER instead of Re-Attention | few-shot state classification · CALVIN ID | accuracy: 0.959 -> 0.945 | Hurt |
| Use PHIER instead of RelViT | few-shot state classification · BEHAVIOR ID | accuracy: 0.866 -> 0.859 | Hurt |
| Use PHIER instead of Re-Attention | state classification · CALVIN novel predicates | accuracy: 0.612 -> 0.863 | Helped |
| Use PHIER instead of RelViT | state classification · BEHAVIOR novel predicates | accuracy: 0.676 -> 0.807 | Helped |
| Use PHIER instead of GPT-4V | zero-shot sim-to-real state classification · BEHAVIOR Vision Suite, all | accuracy: 0.712 -> 0.608 | Hurt |
| Add object-centric encoder (ablation) | PHIER supervised base · CALVIN OOD | accuracy: 0.473 -> 0.627 | Helped |
| Add object-centric encoder (ablation) | PHIER supervised base · BEHAVIOR OOD | accuracy: 0.516 -> 0.653 | Helped |
| Add predicate triplet loss (ablation) | PHIER with object-centric encoder · CALVIN OOD | accuracy: 0.627 -> 0.758 | Helped |
| Add predicate triplet loss (ablation) | PHIER with object-centric encoder · BEHAVIOR OOD | accuracy: 0.653 -> 0.735 | Helped |
| Add norm regularization loss (ablation) | PHIER with predicate triplet loss · CALVIN OOD | accuracy: 0.758 -> 0.823 | Helped |
Surprising findings
- PHIER underperforms Re-Attention in-distribution on CALVIN (0.945 vs 0.959), attributed to constrained representations limiting memorization.
- Pretrained large VLMs (GPT-4V, BLIP-2) remain stronger than PHIER on real-world transfer, with GPT-4V leading by 10.4 points zero-shot.
Limitations
- Relies on LLM accuracy to infer pairwise predicate relations and hierarchy; language-only cues may miss visual ones.
- Hierarchy is softly enforced via losses rather than hard constraints in the forward pass.
- Lower in-distribution accuracy than some baselines on CALVIN due to constrained representations.
Experimental setup
Datasets: CALVIN (simulated) BEHAVIOR (simulated) BEHAVIOR Vision Suite (real-world)
Benchmarks: CALVIN state classification BEHAVIOR state classification BEHAVIOR Vision Suite
Metrics: binary state classification accuracy
Baselines: Re-Attention CoarseFine BUTD RelViT CLIP FiLM SORNet GPT-4V BLIP-2 ViperGPT
Builds on: MaskCLIP Hyperbolic Neural Networks Re-Attention RelViT SORNet
Closest papers by meaning
- STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning (ICLR 2025 · similarity 0.85)
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning (ICLR 2025 · similarity 0.85)
- JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics (AAAI 2026 · similarity 0.85)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.85)
- RoboGround: Robotic Manipulation with Grounded Vision-Language Priors (CVPR 2025 · similarity 0.85)
- SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.