Research map / 3D vision / 3D generation
Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T2Mo, a feed-forward diffusion (DiT, rectified flow) model that generates per-vertex displacements for a static mesh conditioned on sparse 3D trajectories and text, using a shape-grounded trajectory embedding over anchor points.
Problem
Text alone is too ambiguous to specify precise motion for dynamic 3D shape generation, and existing feed-forward dynamic 3D generators offer little explicit spatial control over how object points move.
Approach
Uses 3D trajectories as direct spatial control and maps arbitrarily configured trajectory sets into a fixed-size set of shape-aware anchor tokens (trajectory anchors plus farthest-point-sampled anchors with a learnable null embedding), injected via gated cross-attention into a DiT that also takes text and shape tokens.
Key results
- User study prompt alignment 4.507 for T2Mo vs 2.657 for best text-based baseline (BiMotion) on a 5-point scale.
- 3D trajectory error ADE 0.089 vs 0.605 for the best cascaded video baseline (Tora + Motion 3-to-4), roughly 5-10x lower.
- Ours achieves AMD 0.147 and VBench dynamic degree 0.743 versus 0.046 and 0.329 for BiMotion, but slightly lower motion smoothness (0.984 vs 0.992 for AnimateAnyMesh).
Surprising findings
- Per-trajectory embedding without shape grounding produces motion in regions not specified by the trajectories, and a variant reaches higher dynamic degree at the cost of worse trajectory alignment.
- Baselines' higher motion smoothness is attributed to near-static motion rather than better quality.
Limitations
- Assumes a fixed mesh topology across frames, so splitting, merging, or emerging components are out of scope.
- Quantitative evaluation uses only 30 meshes and user study uses 10 samples per baseline group.
- Slightly lower motion smoothness than the text-based baselines.
Experimental setup
Datasets: BiMO Objaverse Sketchfab
Benchmarks: VBench user study AMD ADE/FDE trajectory error
Metrics: overall consistency aesthetic quality motion smoothness dynamic degree AMD ADE FDE prompt alignment motion plausibility motion expressiveness
Baselines: AnimateAnyMesh BiMotion Tora + ActionMesh Tora + Motion 3-to-4
Builds on: DiT rectified flow AnimateAnyMesh BiMotion Tora
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.87)
- Learning Continuous 3D Words for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions (CVPR 2025 · similarity 0.87)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.87)
- ArtFormer: Controllable Generation of Diverse 3D Articulated Objects (CVPR 2025 · similarity 0.87)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.