Research map / Image generation / Editing and style transfer
General Articulated Objects Manipulation in Real Images via Part-Aware Diffusion Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PA-Diffusion, a training-free part-aware diffusion editing method that manipulates articulated objects in real images via abstract 3D primitive models and dynamic feature maps (manipulated inverted noise, compositional activations) with TCSL and SCSL losses, built on Stable Diffusion v1-5 with T2I-Adapter.
Problem
Editing articulated objects (e.g., opening a laptop or drawer) in real images with text or interaction while avoiding artifacts, blur, and failures on rotated, deformed, or novel-view parts.
Approach
Represent articulated objects as abstract 3D models built from a small library of primitive prototypes, manipulate them in 3D, and use the rendered part masks and sketches to build dynamic feature maps (manipulated inverted noise maps and compositional activation maps/images) that transfer seen-part appearance to new locations while generating novel parts in a consistent style, with texture and st
Key results
- On the 660-image testbench, PA-Diffusion achieves average FID 1.09 versus 1.82 for MasaCtrl and 10.1 for Imagic, a reported improvement of over 40%.
- Ablation shows TCSL and SCSL together reduce average FID from 1.72 (no losses) to 1.09, a 36.6% improvement.
- Training a 3D understanding model with edited samples added to InternetVideo raises performance over the baseline, e.g., mixed-set bbox+axis(rot) accuracy 66.1 vs. baseline reported in Table 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PA-Diffusion instead of MasaCtrl (with T2I Adapter) | Articulated object image editing · FID testbench average | FID: 1.82 -> 1.09 | Helped |
| Add Texture Consistency Score Loss (TCSL) (ablation) | PA-Diffusion · Articulated object image editing | FID: 1.72 -> 1.10 | Helped |
| Add Style Consistency Score Loss (SCSL) alongside TCSL (ablation) | PA-Diffusion · Articulated object image editing | FID: 1.10 -> 1.09 | Helped |
| Add PA-Diffusion edited images to InternetVideo training data | InternetVideo-trained articulated object understanding model · InternetVideo test set | bbox: 62.1 -> 66.1 | Helped |
| Use the full rather than half training set | Articulated object understanding model · Articulated object understanding | normal < 30°: 54.1/63.2 | Helped |
Surprising findings
- The method also handles non-rigid objects (broken cup, shark toy) and unconventional joints, which the authors did not expect to work well.
Limitations
- Output quality depends heavily on input image quality; blurry or low-resolution inputs degrade DDIM inversion and edits.
- Large deformation or movement of objects causes failure because manipulating inverted noise disturbs its distribution.
- Deformable objects and fluids remain challenging.
Experimental setup
Datasets: InternetVideo dataset (for downstream 3D articulated object understanding) 660 real images collected from the web (testbench)
Benchmarks: articulated object manipulation testbench (660 real images, 6 categories) InternetVideo test set (6,231 real images)
Metrics: FID BBox IoU accuracy axis EA score surface normal error under 30 degrees
Baselines: Imagic DragDiffusion MasaCtrl Image Sculpting
Builds on: DDIM inversion T2I-Adapter MasaCtrl Grounded Segment Anything Image Sculpting
Closest papers by meaning
- SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects (ICLR 2025 · similarity 0.89)
- CAGE: Controllable Articulation GEneration (CVPR 2024 · similarity 0.88)
- AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning (ICLR 2025 · similarity 0.88)
- DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data (NeurIPS 2025 · similarity 0.87)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.87)
- Object Pose Estimation via the Aggregation of Diffusion Features (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.