Research map / Vision-language and multimodal / Robot action and navigation
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a DINOv2-based image-only encoder by aligning image transitions, language, and 3D flow via a simplex (triangle-area) energy with cosine regularization and InfoNCE-style contrastive loss, yielding dynamics-aware visual backbones for MLP, diffusion, and VLA policies.
Problem
Visual encoders pre-trained for static recognition or vision-language alignment lack motion/dynamics awareness, so they attend to control-irrelevant regions and limit robot manipulation generalization.
Approach
Use image transitions, language, and 3D flow from action-free human and robot videos as training-time supervision; minimize the triangle area they span in a shared hyperspherical space, fix geometric ambiguity with a cosine regularizer on language-flow pairs, and prevent collapse with an InfoNCE-style contrastive objective plus temporal contrastive and actor (flow prediction) losses.
Key results
- On LIBERO with frozen encoders and Diffusion Policy, DynaFLIP reaches 41.5% mean success vs 37.2% for the best baseline (DINOv2); with LoRA fine-tuning it reaches 81.0% vs 79.1% (SigLIP).
- On MetaWorld (15 tasks, frozen MLP), DynaFLIP achieves 78.9% mean vs 76.0% for LIV; on RLBench it achieves 54.0% vs 48.6% for LIV.
- Real-world OOD gains reach up to +22.5% over the strongest baseline per the abstract.
Surprising findings
- Removing the contrastive negative tuples drops LIBERO mean success from 44.0 to 18.1, the largest ablation drop, showing naive volume minimization collapses.
Limitations
- Pre-training uses only 260K trajectories, smaller than data scales of some large baselines.
- 3D flow uses a uniform 20x20 grid, so task-irrelevant motion can inject noisy supervision.
- Real-world results are on a single UR3 setup with three tasks.
Experimental setup
Datasets: AgiBot Droid Open X-Embodiment BridgeData V2 Ego4D Something-Something V2 MetaWorld RLBench LIBERO
Benchmarks: MetaWorld RLBench LIBERO Real-world UR3 manipulation
Metrics: success rate (%) control-relevant score S_m
Baselines: R3M VC-1 LIV DINOv2 CLIP SigLIP
Builds on: R3M TraceForge SimplexCLIP-style simplex alignment DINOv2 InfoNCE Diffusion Policy
Closest papers by meaning
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.91)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.90)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.89)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.89)
- FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.