Research map / Vision-language and multimodal / Robot action and navigation
Selective Visual Representations Improve Convergence and Generalization for Embodied AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a small task-conditioned codebook bottleneck (K=256 codes, D_c=10) on top of EmbCLIP, trained end-to-end with PPO, to filter visual embeddings for object navigation and displacement, improving success, episode length, and trajectory smoothness across five simulators.
Problem
General-purpose visual representations (e.g., CLIP) fed to embodied AI policies contain much task-irrelevant information, which adds noise to learning and distracts agents from task-relevant cues.
Approach
A learnable codebook placed after goal-visual fusion produces a softmax distribution over K latent codes, takes a convex combination to form a compact bottleneck, and upsamples it back to the policy input; it is trained jointly with the policy using only task reward, with dropout on code probabilities to prevent codebook collapse.
Key results
- On ProcTHOR-10k validation, EmbCLIP-Codebook raises success rate from 67.70% to 73.72% and lowers episode length from 182 to 136 versus EmbCLIP.
- On AI2-iTHOR zero-shot, success rate improves from 70.00% to 78.40%; on ManipulaTHOR, m-VOLE+Codebook reaches 65.10% SR vs 59.60%.
- In Habitat finetuning of only the adaptation module, SR rises from 36.45% to 50.25% over EmbCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the goal-conditioned codebook bottleneck | EmbCLIP · ProcTHOR-10k validation | SR (%): 67.70 -> 73.72 | Helped |
| Add the goal-conditioned codebook bottleneck | EmbCLIP · ArchitecTHOR 0-shot | SR (%): 55.80 -> 58.33 | Helped |
| Add the goal-conditioned codebook bottleneck | EmbCLIP · RoboTHOR 0-shot | SR (%): 51.32 -> 55.00 | Helped |
| Add the goal-conditioned codebook bottleneck | EmbCLIP · AI2-iTHOR 0-shot | SR (%): 70.00 -> 78.40 | Helped |
| Add the goal-conditioned codebook bottleneck | m-VOLE · ManipulaTHOR | SR (%): 59.60 -> 65.10 | Helped |
| Fine-tune only the adaptation module with codebook representations | EmbCLIP-Codebook · Habitat HM3D Semantics | SR (%): 36.45 -> 50.25 | Helped |
| Fine-tune the entire model with codebook representations | EmbCLIP-Codebook · Habitat HM3D Semantics | SR (%): 58.00 -> 55.00 | Hurt |
| Add the goal-conditioned codebook bottleneck | DINOv2 · ProcTHOR-10k validation | SR (%): 74.25 -> 76.31 | Helped |
| Add the goal-conditioned codebook bottleneck | DINOv2 · RoboTHOR 0-shot | SPL: 29.36 -> 28.01 | Hurt |
| Replace the codebook with an autoencoder bottleneck (ablation) | EmbCLIP · ProcTHOR-10k validation, 210M steps | SR (%): 72.49 -> 59.35 | Hurt |
| Replace the codebook with self-attention (ablation) | EmbCLIP · ProcTHOR-10k validation, 210M steps | SR (%): 72.49 -> 56.50 | Hurt |
| Remove dropout regularization from codebook probabilities (ablation) | EmbCLIP-Codebook · ProcTHOR-10k validation | SR (%): 62.52 -> 71.39 | Hurt |
Surprising findings
- The codebook lowers SPL on several benchmarks (e.g., ProcTHOR 49.00 to 48.37) even while success and episode length improve, which the authors attribute to SPL ignoring rotations and look actions.
- Linear probing shows the codebook reduces accuracy for predicting presence of all 125 object categories and 16 goal categories (F1 drops from 0.53 to 0.11 for goal presence), while improving goal visibility and distance prediction.
Limitations
- Performance is bounded by the perceptual capability of the frozen pretrained encoder, e.g., small or hard-to-identify goals.
- SPL metric disagrees with the proposed SEL and episode-length gains, so efficiency claims depend on the chosen metric.
- Some ablations (e.g., SelfAttn, AE) were run only to 210M steps due to compute and time limits.
Experimental setup
Datasets: ProcTHOR-10k APND Habitat 2022 ObjectNav (HM3D Semantics)
Benchmarks: ProcTHOR-10k ArchitecTHOR RoboTHOR AI2-iTHOR ManipulaTHOR Habitat 2022 ObjectNav
Metrics: success rate episode length curvature SPL SEL pickup success rate linear probing accuracy F1
Baselines: EmbCLIP m-VOLE DINOv2 baseline SGC NRC EmbCLIP-AE EmbCLIP-SelfAttn
Builds on: EmbCLIP CLIP PPO DD-PPO m-VOLE
Closest papers by meaning
- Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control (NeurIPS 2024 · similarity 0.90)
- DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning (ICML 2024 · similarity 0.87)
- Task-Conditioned Adaptation of Visual Features in Multi-Task Policy Learning (CVPR 2024 · similarity 0.87)
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.87)
- Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation (NeurIPS 2025 · similarity 0.87)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.