Research map / Vision-language and multimodal / Robot action and navigation

$\textit{HiMaCon:}$ Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data

NeurIPS 2025 · Vision-language and multimodal › Robot action and navigation · approach: Memory and subgoal-based manipulation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HiMaCon, a self-supervised method that learns hierarchical manipulation concept latents from unlabeled multi-modal robot demonstrations via cross-modal mask-and-predict and multi-horizon future prediction, then uses them as an auxiliary prediction target for ACT and Diffusion Policy on LIBERO and a real Mobile ALOHA robot.

Problem

Robot manipulation policies fail to generalize to novel objects, obstacles, and environments; the paper seeks transferable, hierarchical representations of manipulation patterns learned without human annotation.

Approach

Learn continuous concept latents with a cross-modal correlation network (mask-and-predict over modality bipartitions, a conditional-MI proxy) and a multi-horizon future predictor conditioned on a coherence threshold epsilon that segments demonstrations into sub-processes; then train policies to jointly predict actions and concepts.

Key results

• With ACT on LIBERO-90, the method reaches 74.8% success vs 73.4% for the best baseline (XSkill) and 46.6% for plain imitation learning.
• With Diffusion Policy on LIBERO-90, the method reaches 89.6% vs 87.7% for XSkill and 75.1% for plain DP; on LIBERO-GOAL (unseen environment) DP reaches 95.7%.
• Real-world Mobile ALOHA experiments with six generalization shifts report consistent gains over baselines, especially on obstacles and barriers.

Measured effects reported in the paper

Surprising findings

• Uniform sampling of the coherence threshold epsilon outperforms sparse and biased sampling strategies, with the biased range giving the lowest scores.
• The ACT ablation shows the best concept-decoder layer and loss weight are not monotonic, with lambda_mc=1.0 at L=2 best for ACT but lambda_mc=0.1 at L=1 best for DP.

Limitations

• Inference-time concept labeling strategy (using future context windows) is acknowledged as not optimal and left for future work.
• Hierarchy derivation relies on a threshold-based method presented as a proof of concept; alternatives are not explored.
• Hyperparameters such as the concept decoder layer L and loss weight lambda_mc are selected by tuning on the evaluation tasks, and the real-world results are described qualitatively in the text without reported success rates in the provided extract.

Experimental setup

Datasets: LIBERO-90 LIBERO-LONG LIBERO-GOAL BridgeDataV2

Benchmarks: LIBERO-90 LIBERO-LONG LIBERO-GOAL Mobile ALOHA cup cleaning (real-world)

Metrics: success rate (%) conditional mutual information sub-goal similarity

Baselines: InfoCon XSkill DecisionNCE RPT CLIP DINOv2 Plain imitation learning Next-step prediction All-masked variant

Builds on: Diffusion Policy ACT InfoCon XSkill RPT

Closest papers by meaning

• InfoCon: Concept Discovery with Generative and Discriminative Informativeness (ICLR 2024 · similarity 0.91)
• Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.87)
• RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.87)
• Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.87)
• Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following (NeurIPS 2025 · similarity 0.87)
• AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.86)