Research map / Image generation / Text-to-image diffusion
PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents PIA, a plug-and-play personalized image animator that adds a trainable condition module (conditioned on the condition frame and inter-frame affinity) to a frozen Stable Diffusion with AnimateDiff temporal layers, and introduces the AnimateBench benchmark.
Problem
Animating personalized text-to-image outputs while preserving their style and fine details, and keeping motion controllable by text prompts, which existing temporal-alignment-based T2V adaptations struggle with.
Approach
Insert a lightweight zero-initialized condition module into the U-Net input convolution that takes the condition frame latent concatenated with a per-frame inter-frame affinity map (from HSV L1 distances on training videos); train it jointly with temporal layers so the temporal layers focus on motion, then swap in any personalized T2I model at inference.
Key results
- PIA reaches an AnimateBench image CLIP score of 93.44 and text CLIP score of 68.74, versus 89.72/68.41 for AnimateDiff.
- In a user study PIA is preferred 0.525 for image alignment and 0.670 for text alignment, the highest among compared methods.
- Ablation: inter-frame affinity gives image CLIP score 225.9 versus 210.9 for one-hot affinity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PIA instead of VideoComposer | PIA · AnimateBench | image: 90.10 -> 93.44 | Helped |
| Use PIA instead of AnimateDiff | PIA · AnimateBench | text: 68.41 -> 68.74 | Helped |
| Use PIA instead of VideoComposer | PIA · WebVid | image: 85.03 -> 85.11 | Helped |
| Use PIA instead of VideoComposer | PIA · WebVid | text: 75.52 -> 76.59 | Helped |
| Use PIA instead of AnimateDiff | PIA · User Study | image: 0.295 -> 0.525 | Helped |
| Use PIA instead of AnimateDiff | PIA · User Study | text: 0.220 -> 0.670 | Helped |
| Replace one-hot frame affinity with inter-frame affinity (ablation) | PIA · AnimateBench | image CLIP score: 210.9 -> 225.9 | Helped |
Surprising findings
- Color discrepancy appears when the input image style is far from WebVid's real-world training domain.
- Missing trigger words in animation prompts can cause significant color inconsistencies.
Limitations
- Color shifts when input images differ strongly from the training domain (WebVid real-world videos).
- Color inconsistency when personalization trigger words are omitted from animation prompts.
- Training domain gap: WebVid has motion blur and compression artifacts, limiting stylized animation quality.
Experimental setup
Datasets: WebVid
Benchmarks: AnimateBench WebVid
Metrics: CLIP score image alignment CLIP score text alignment user preference rate
Baselines: VideoComposer AnimateDiff Pika Labs Gen-2
Builds on: Stable Diffusion AnimateDiff ControlNet
Closest papers by meaning
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR 2024 · similarity 0.89)
- PersonaBooth: Personalized Text-to-Motion Generation (CVPR 2025 · similarity 0.85)
- Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · similarity 0.85)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.85)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.85)
- MotiF: Making Text Count in Image Animation with Motion Focal Loss (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.