Research map / Label-efficient and robust learning / Recognition and long-tail
InfoCon: Concept Discovery with Generative and Discriminative Informativeness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoCon, a self-supervised VQ-VAE-based method that discovers manipulation concepts from unlabeled demonstrations via generative and discriminative informativeness losses, and uses them as key-state guidance for CoTPC policies.
Problem
Discovering manipulation concepts and sub-goal (key) states from noisy, unlabeled robot demonstrations without human annotation or language descriptions, so that concepts are grounded in physical states.
Approach
Models manipulation concepts as discrete codebook symbols that act as both generative goals (predicting the future key state via mutual information) and discriminative goals (a learned compatibility function whose gradient should predict the next action), integrated into a VQ-VAE partitioning network.
Key results
- InfoCon-guided CoTPC reaches 96.6% (seen) and 81.0% (unseen) success on P&P Cube, outperforming AWE (96.2/78.0) and LLM+CLIP and matching or exceeding GT key states in most settings.
- On Peg Insertion, InfoCon achieves 63.6% seen and 17.8% unseen success, above GT key states (52.8/16.8) and AWE (31.6/4.3).
- On Turn Faucet unseen-faucet setting, InfoCon (17.8%) trails GT key states (31.0%), so it does not beat the oracle everywhere.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use InfoCon-discovered key states instead of the strongest listed baseline key-state method | CoTPC manipulation policies · P&P Cube, Stack Cube, and Peg Insertion, seen and unseen; Turn Faucet, all reported settings | success rate (%): P&P Cube seen: 96.2 -> 96.6; Turn Faucet s.&sf.: 56.4 -> 53.8 | Mixed |
| omit discriminative goal losses (use generative goal loss only) (ablation) | InfoCon · P&P Cube, Stack Cube, Turn Faucet, and Peg Insertion | success rate (%): P&P Cube seen: 96.6 -> 69.8; Stack Cube seen: 63.0 -> 46.8; Peg Insertion unseen: 17.8 -> 5.5 | Hurt |
| omit generative goal loss (use discriminative goal losses only) (ablation) | InfoCon · P&P Cube, Stack Cube, Turn Faucet, and Peg Insertion | success rate (%): P&P Cube seen: 96.6 -> 72.0; Peg Insertion seen: 63.6 -> 64.2; Peg Insertion unseen: 17.8 -> 10.5 | Mixed |
Surprising findings
- Ablation shows generative-only or discriminative-only losses both degrade policy success, especially in unseen environments.
- HIS human-intuition similarity correlates only weakly with policy performance, suggesting discovered concepts need not match human semantics.
Limitations
- Performance on unseen faucet types remains low (17.8%) and below GT key states on Turn Faucet.
- Concept count is a fixed hyperparameter (max 10) and concept activation is task-dependent; over-smoothing needed a reconstruction regularizer.
- Evaluated only on four simulated ManiSkill2 tasks; semantic naming of concepts is qualitative on Peg Insertion only.
Experimental setup
Datasets: ManiSkill2 demonstrations (P&P Cube, Stack Cube, Turn Faucet, Peg Insertion)
Benchmarks: ManiSkill2 P&P Cube ManiSkill2 Stack Cube ManiSkill2 Turn Faucet ManiSkill2 Peg Insertion
Metrics: success rate (%) Human Intuition Similarity (HIS)
Baselines: Vanilla BC Decision Transformer Behavior Transformer Decision Diffuser MaskDP Last State AWE LLM+CLIP GT Key States
Builds on: VQ-VAE CoTPC AWE Decision Transformer
Closest papers by meaning
- $\textit{HiMaCon:}$ Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data (NeurIPS 2025 · similarity 0.91)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.87)
- RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.