Research map / Video / Video generation

PAI-Actor: Cinematic Multi-Actor Character Replacement in Dynamic Scenes

NeurIPS 2026 · Video › Video generation · approach: Trajectory-guided motion control

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAI-Actor, a movie-driven framework for multi-character replacement in real video that trains a bidirectional in-context diffusion transformer teacher and distills it into a causal autoregressive video-to-video student for long 1080P generation.

Problem

Replacing and animating multiple characters inside real dynamic movie clips at high resolution while preserving background, camera motion, interactions, and lighting, and scaling to long videos.

Approach

Cast character replacement as structure-guided human recovery: mask original performers, overlay skeletons/face landmarks on the masked video, and train the model to recover the performers from reference images using real movie clips; then distill the bidirectional in-context teacher into a block-causal autoregressive student with two-stage training (causal adaptation and on-policy self-forcing) f

Key results

• Ours-Bidir achieves MSE 1.89 (single) vs 3.22 for the next best VACE on the Character-Animation subset, and best LPIPS 0.114 and Pose-Match 0.871 among compared methods.
• Ours-Causal on ~417-frame long videos keeps competitive metrics (e.g., Pose-Match 0.856 single, BG Cons. 9.88 single).
• Ours-Bidir achieves the highest LLM-judged BG Cons., FG-BG Fusion and Char. Cons. on the Character-Replacement subset (8.36 BG Cons. single/multi).

Surprising findings

• MultiAnimate reports higher SSIM than the proposed method, attributed to foreground-only evaluation.
• Causal student without self-forcing can show facial degradation and struggles with out-of-distribution styles such as 2D anime.

Limitations

• Requires many denoising steps, so inference latency is high (Ours-Bidir ~0.06 FPS).
• Scale mismatch between reference and driving view and severe occlusions/overlaps cause face distortion and ambiguous poses.
• Causal student struggles with non-photorealistic styles such as 2D anime.

Experimental setup

Datasets: PAI-Actor Dataset

Benchmarks: PAI-Benchmark

Metrics: MSE SSIM LPIPS Pose-Dist Pose-Match Motion Cons. Expr.-LMD BG Cons. FG-BG Fusion Illum. Harmony

Baselines: Kling O1 Domo Viggle Wan-Animate VACE MoCha MultiAnimate SCAIL SCAIL-2

Builds on: Wan2.2-TI2V-5B rectified flow self-forcing LoRA

Closest papers by meaning

• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.87)
• MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.87)
• EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration (NeurIPS 2026 · similarity 0.86)
• Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
• Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.85)
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (CVPR 2024 · similarity 0.85)