Research map / Video / Video generation
SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SocialDirector, a training-free cross-attention controller (social actor masking plus directional reweighting) that improves who-does-what-to-whom control in multi-person image-to-video DiTs, and a VLM-majority-vote evaluation benchmark of 149 clips.
Problem
Existing multi-person video generators lack explicit control over who performs which action, when it occurs, and toward whom, causing actor-action mismatch, disordered social dynamics, and wrong targets.
Approach
Mask cross-attention so each person's spatial-temporal visual tokens attend only to their own event text within its time window (social actor masking), and add a positive bias to directional words so actions align with the intended target (directional reweighting), with no training or architectural change.
Key results
- On Wan2.2, SocialDirector raises action accuracy from 72.2 to 76.1, target accuracy from 69.2 to 72.3, and stillness accuracy from 85.1 to 88.2.
- On LTX-2.3, action accuracy rises from 71.6 to 74.3 and target accuracy from 69.4 to 70.8, with video quality roughly unchanged.
- Per-sample overhead is about 1.7% extra time on Wan2.2 and 2.7% on LTX-2.3 with no added FLOPs.
Surprising findings
- Directional reweighting alone gives no improvement over the base model, since attention leakage must be resolved first by actor masking.
- Audio-driven baselines reach stillness scores above the GT oracle because they tend to produce no action at all, inflating the metric.
Limitations
- No audio modality, so speech-grounded behaviors such as lip sync are not handled.
- Evaluated only on short clips; no long-form multi-turn generation.
- Relies on ground-truth bounding boxes and fails under heavy occlusion and large actor movement.
Experimental setup
Datasets: MELD MMSI SocialGesture
Benchmarks: SocialInteraction evaluation set (149 clips)
Metrics: action accuracy target accuracy stillness accuracy FVD LPIPS ViCLIP DOVER video quality DINOv2 ID preservation
Baselines: MultiTalk EchoMimic v3 HunyuanVideoAvatar Bind-Your-Avatar AnyTalker Playmate2 Wan2.2 LTX-2.3
Builds on: diffusion transformer cross-attention layout-guided cross-attention control
Closest papers by meaning
- SVBench: Evaluation of Video Generation Models on Social Reasoning (CVPR 2026 · similarity 0.88)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.84)
- ReGenNet: Towards Human Action-Reaction Synthesis (CVPR 2024 · similarity 0.84)
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.84)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.84)
- Action Imitation in Common Action Space for Customized Action Image Synthesis (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.