Research map / 3D vision / 3D generation
ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASHiTA, which alternates LLM-assisted hierarchical task analysis with task-driven 3D scene graph construction (via a hierarchical Information Bottleneck) to ground high-level tasks into subtasks and items in a scene.
Problem
Grounding abstract, high-level natural-language tasks to a 3D scene requires environment-dependent decomposition into subtasks and items, which existing scene graph and LLM planning methods do not handle.
Approach
Generalizes the Information Bottleneck to a hierarchical form (H-IB) that builds a scene graph aligned with a task hierarchy, and alternates this bottom-up/top-down scene construction with LLM-based task refinement and spatial grounding, so the task breakdown and the scene representation are co-determined.
Key results
- On SG3D HM3DSem with GT segmentation, ASHiTA with text embeddings reaches 65.4% subtask accuracy versus 25.3% for 3D-VisTA.
- On the incremental 8-scene setting, ASHiTA achieves 21.7% subtask accuracy versus 8.98% for HOV-SG + GPT.
- In hierarchical task analysis, ASHiTA reaches 10.39% subtask recall and 9.27% task accuracy, outperforming Hydra + GPT (9.43%, 4.88%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ASHiTA instead of zero-shot visual-grounding baselines | SG3D HM3DSem grounding with ground-truth 3D instance segmentation · SG3D HM3DSem | s-acc: 25.3 -> 28.71; 24.4 -> 28.71 | Helped |
| use ASHiTA instead of Hydra + GPT or HOV-SG | sequential grounding on incrementally built scene graphs · SG3D HM3DSem, 8 scenes | s-acc: 8.18 -> 21.7; 8.98 -> 21.7 | Helped |
| use ASHiTA instead of Hydra + GPT or HOV-SG + GPT for task hierarchy generation | SG3D HM3DSem task hierarchy analysis · SG3D HM3DSem, 8 RGB-D sequences | t-acc: 4.88 -> 9.27; 1.95 -> 9.27 | Helped |
| replace recursively running original IB with H-IB (ablation) | ASHiTA · SG3D HM3DSem, 8 RGB-D sequences | s-rec: 1.51 -> 10.39 | Helped |
| replace recursively running original IB with H-IB (ablation) | ASHiTA · SG3D HM3DSem, 8 RGB-D sequences | s-prec: 24.53 -> 20.6 | Hurt |
| add Top-Down pruning (ablation) | ASHiTA · SG3D HM3DSem, 8 RGB-D sequences | t-acc: 5.37 -> 9.27 | Helped |
| add Spatial Update (ablation) | ASHiTA · SG3D HM3DSem, 8 RGB-D sequences | t-acc: 6.34 -> 9.27 | Helped |
| add LLM-based Hierarchy Refinement (ablation) | ASHiTA · SG3D HM3DSem, 8 RGB-D sequences | t-acc: 6.83 -> 9.27 | Helped |
Surprising findings
- Recursive IB yields higher subtask precision (24.53%) but much lower recall (1.51%), showing the hierarchical coupling matters for recall.
Limitations
- Scene graph lacks relations, so spatial specifications like 'inside' or 'on top of' cannot be handled.
- Task hierarchy granularity is limited by the word bank; similar objects are not differentiated.
- No guarantee the generated subtasks are sufficient to complete the high-level task; tree structure prevents sharing objects across subtasks.
Experimental setup
Datasets: SG3D HM3DSem 8 RGB-D sequences from HOV-SG
Benchmarks: SG3D HM3DSem 8 HM3DSem RGB-D sequences (205 tasks)
Metrics: subtask accuracy task accuracy subtask recall subtask precision
Baselines: 3D-VisTA PQ3D Hydra + GPT HOV-SG GPT w/ GT labels Recursive IB Primitives + GPT
Builds on: Information Bottleneck Clio Hydra HOV-SG SG3D
Closest papers by meaning
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.88)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.88)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.87)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.87)
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.87)
- TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.