Research map / Vision-language and multimodal / Robot action and navigation
Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses intermediate U-Net features from pre-trained Stable Diffusion (optionally fine-tuned on robot/human video frames) as frozen visual representations for control policies, competitive with CLIP, VC-1, and R3M across manipulation, navigation, and grounding tasks.
Problem
Contrastively trained vision-language representations such as CLIP give fine-grained scene understanding that is insufficient for embodied control; text-to-image diffusion model representations have not been explored for control.
Approach
Extract representations from a partially noised latent passed through Stable Diffusion's U-Net, conditioned on the task text, aggregating multi-layer (mid and down-block) features; optionally fine-tune the U-Net on egocentric/robot video frames with the original generative objective, and use cross-attention maps as category-agnostic inputs for open-vocabulary navigation.
Key results
- SCR-ft reaches 41.9% success on OVMM Gaze and SCR-ft-attn reaches 43.6%, versus 38.7% for CLIP and 40.6% for VC-1.
- On ImageNav, SCR reaches 73.9% success, above VC-1 (70.3%) and MVP (68.1%); SCR-ft reaches 69.5%.
- On Meta-World, SCR-ft achieves 94.9% success, and on Franka-Kitchen SCR-ft achieves 49.9% versus 36.3% for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SCR on general robotics datasets (ablation) | SCR · Franka-Kitchen | success rate: 45.0 -> 49.9 | Helped |
| Fine-tune SCR on general robotics datasets (ablation) | SCR · Meta-World | success rate: 94.4 ± 1.9 -> 94.9 ± 2.0 | No clear effect |
| Add cross-attention maps to SCR-ft (ablation) | SCR-ft · OVMM | success: 41.9 ± 1.0 -> 43.6 ± 2.1; +1.7% | Helped |
| Use timestep 0 instead of timestep 100 (ablation) | SCR · Franka-Kitchen | success rate: 42.0 ± 3.7 -> 49.9 ± 3.4 | Helped |
| Use timestep 0 instead of timestep 200 (ablation) | SCR · Franka-Kitchen | success rate: 35.1 ± 3.2 -> 49.9 ± 3.4 | Helped |
| Use timestep 0 instead of timestep 10 (ablation) | SCR · Franka-Kitchen | success rate: 48.2 ± 3.1 -> 49.9 ± 3.4 | No clear effect |
| Use outputs from multiple U-Net layers rather than Down[1-3] alone (ablation) | SCR · Franka-Kitchen | success rate: 43.0 ± 3.4 -> 49.9 ± 3.4 | Helped |
| Add the downsampling block features to mid-block features (ablation) | SCR · Franka-Kitchen | success rate: 41.6 ± 3.3 -> 49.9 ± 3.4 | Helped |
| Add a relevant prompt during representation extraction (ablation) | SCR · Franka-Kitchen | success rate: 49.9 ± 3.4 -> 49.2 ± 3.5 | No clear effect |
| Add an irrelevant prompt during representation extraction (ablation) | SCR · Franka-Kitchen | success rate: 49.9 ± 3.4 -> 48.7 ± 3.3 | Hurt |
| Use a compression layer instead of MAP/CLS aggregation (ablation) | VC-1 · Grasp Affordance Prediction | average precision: 24.7 -> 83.9 | Helped |
| Use a compression layer instead of MAP/CLS aggregation (ablation) | CLIP · Referring Expressions Grounding | bounding box accuracy: 68.1 -> 94.3 | Helped |
Surprising findings
- Fine-tuning CLIP with the contrastive objective did not improve Franka-Kitchen performance (36.9 to 34.2), while the diffusion model fine-tuning gave a 4% gain.
- Providing task-relevant or irrelevant text prompts to the U-Net did not help and sometimes slightly degraded Franka-Kitchen performance.
Limitations
- Representation size is very large (about 100k-dimensional features before compression), requiring a forward pass through about 400M parameters.
- Fine-tuning did not improve fine-grained visual prediction tasks, suggesting a gap between visual prediction and control benchmarks.
- Results on some baselines depend on released architectures, so parameter counts could not be fully matched.
Experimental setup
Datasets: EpicKitchens Something-Something-v2 Bridge-v2 Meta-World RoboHive Franka-Kitchen Habitat ImageNav Gibson episodes OVMM Habitat episodes OCID-Ref Grasp affordance dataset
Benchmarks: Meta-World Franka-Kitchen ImageNav OVMM Gaze OCID-Ref referring expression grounding Grasp affordance prediction
Metrics: success rate bounding box accuracy at 25% IoU precision at top 90/95/99 percentile
Baselines: R3M CLIP VC-1 MVP Voltron SD-VAE LIV Detic
Builds on: Stable Diffusion CLIP DD-PPO CortexBench VC-1
Closest papers by meaning
- Selective Visual Representations Improve Convergence and Generalization for Embodied AI (ICLR 2024 · similarity 0.90)
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.88)
- TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models (ICLR 2026 · similarity 0.88)
- MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.87)
- DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning (ICML 2024 · similarity 0.87)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.