Research map / Video / Video generation
Action Imitation in Common Action Space for Customized Action Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TwinAct, which builds a PCA-based common action space from 832 action phrases and imitates customized actions by MLP-predicted coefficients over action bases, trained with reconstruction plus action similarity loss to generate few-shot customized action images.
Problem
Few-shot customization of text-guided diffusion models for actions fails because action tokens entangle action with actor appearance; existing methods lack an inductive bias to decouple them.
Approach
Constructs a common action space from PCA over CLIP text embeddings of curated action phrases, and represents the customized action token as a linear combination of these action bases with coefficients predicted by an MLP from a CLIP action encoder; adds an action similarity loss computed on predicted denoised images.
Key results
- TwinAct achieves SAction 69.47 and SActor 73.34, versus best baseline ADI at 45.32 and 48.76.
- User study ratings: UAction 3.86 and UActor 4.12 for TwinAct, versus 2.25 and 2.08 for ADI (1-5 scale).
- Ablation: PCA components p=512 gives best results; removing the action space drops SActor to 58.86 and SAction to 52.18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TwinAct instead of ADI | Customized action generation · Customized action benchmark | SAction: 45.32 -> 69.47 | Helped |
| use TwinAct instead of ADI | Customized action generation · Customized action benchmark | SActor: 48.76 -> 73.34 | Helped |
| use TwinAct instead of ADI | Customized action generation · Customized action benchmark | UAction: 2.25 -> 3.86 | Helped |
| use TwinAct instead of ADI | Customized action generation · Customized action benchmark | UActor: 2.08 -> 4.12 | Helped |
| add the common action space (ablation) | TwinAct · Customized action benchmark | SAction: 52.18 -> 69.47 | Helped |
| use all action phrases instead of 50% (ablation) | TwinAct · Customized action benchmark | SAction: 61.86 -> 69.47 | Helped |
| add the action encoder (ablation) | TwinAct · Customized action benchmark | SAction: 59.06 -> 69.47 | Helped |
| add action similarity loss (ablation) | TwinAct · Customized action benchmark | SAction: 63.58 -> 69.47 | Helped |
| set principal components to p = 64 instead of p = 512 (ablation) | TwinAct · Customized action benchmark | SAction: 54.27 -> 69.47 | Helped |
| set principal components to p = 256 instead of p = 512 (ablation) | TwinAct · Customized action benchmark | SAction: 61.66 -> 69.47 | Helped |
| set principal components to p = 768 instead of p = 512 (ablation) | TwinAct · Customized action benchmark | SAction: 67.63 -> 69.47 | Helped |
Limitations
- Action space may miss some actions, limiting expressiveness of the output domain.
- Inherits physical defects of diffusion models in generating human bodies (e.g., hands, limbs), mitigated only by negative prompts.
- Evaluation benchmark is small (12 actions) and not publicly released.
Experimental setup
Datasets: action phrases generated by GPT-4 (832 after filtering) customized action benchmark (12 actions, ~10 images each)
Benchmarks: customized action benchmark (12 actions)
Metrics: SAction (CLIP score) SActor (face recognition similarity) UAction user score UActor user score
Baselines: Textual Inversion DreamBooth Reversion Custom Diffusion P+ ADI Stable Diffusion ControlNet
Builds on: Textual Inversion DreamBooth Custom Diffusion P+ Reversion
Closest papers by meaning
- Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Diffusion Diffusion Process (ICLR 2026 · similarity 0.87)
- Think Then React: Towards Unconstrained Action-to-Reaction Motion Generation (ICLR 2025 · similarity 0.87)
- ReactID: Synchronizing Realistic Actions and Identity in Personalized Video Generation (ICLR 2026 · similarity 0.86)
- Unifying Diffusion and Autoregression for Generalizable Vision-Language-Action Model (ICLR 2026 · similarity 0.86)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.