Research map / Vision-language and multimodal / Cross-modal learning

MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MegActor-Σ, a mixed-modal diffusion transformer for portrait animation that combines audio and visual control via a three-stage "Modality Decoupling Control" training scheme and an "Amplitude Adjustment" inference strategy, trained only on a filtered 313-hour public dataset.

Problem

Portrait animation methods use either audio or visual control; combining them is hard because weak audio control is overwhelmed by strong visual control (visual leakage into the mouth region), and no flexible amplitude control exists across modalities.

Approach

Replace the UNet backbone with a PixArt-α-based Diffusion Transformer, decouple visual control spatially with face-dropout masks and masked attention/MSE loss, stage-wise mix audio and visual conditions to balance their strength, and scale audio/visual control strength at inference (audio_scale and warp-based motion scaling).

Key results

• On HDTF, MegActor-Σ (1.4B) reaches FID 31.497, FVD 302.87, LPIPS 0.0812 and Sync-C 6.8226, the best of all compared methods, despite training only on public data.
• On CCv2, MegActor-Σ achieves FID 35.118, FVD 345.66 and Sync-C 6.5407, outperforming private-data baselines such as EchoMimic and Hallo.
• In the modality ablation on HDTF, audio+visual control (A+V) beats audio-only and visual-only on every metric (e.g., FVD 302.87 vs 383.87 for A and 313.33 for V).

Measured effects reported in the paper

Surprising findings

• Even when only eye patches are provided as visual control, the mouth region still shows speaking motion, revealing visual leakage that blocks audio control.

Limitations

• Visual leakage and audio-control dominance are only partly resolved; the mouth can still follow visual cues in some cases.
• Evaluation is limited to HDTF and CCv2 with a 9:1 identity split; no human evaluation is reported.

Experimental setup

Datasets: VoxCeleb Talking-Head 1k MultiTalk CCv2 HDTF

Benchmarks: HDTF CCv2

Metrics: FID FVD LPIPS Sync-C Sync-D

Baselines: AniPortrait Hallo MagicDance MegActor Follow-Your-Emoji X-Portrait V-Express EchoMimic

Builds on: PixArt-α MegActor AnimateAnyone MagicAnimate EasyAnimate

Closest papers by meaning

• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.87)
• RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.87)
• HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation (CVPR 2025 · similarity 0.86)
• A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.86)
• Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.86)
• Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.86)