Research map / Video / Video generation
PerReactor: Offline Personalised Multiple Appropriate Facial Reaction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerReactor, a GAN-based offline personalised multiple appropriate facial reaction generator that uses appropriateness and realism discriminators and a personalised behaviour pattern module learned from a listener's historical face video.
Problem
Automatic facial reaction generation suffers from the one-to-many mapping problem, reconstruction losses fail to capture appropriateness and realism, and existing multiple appropriate facial reaction (MAFRG) models ignore listener-specific personalised behaviour patterns.
Approach
Trains the generator adversarially with appropriateness discriminators (generic and personalised) and a realism discriminator so that each speaker behaviour maps to a distribution of appropriate reactions; a Personalised Behaviour Pattern Modelling module disentangles listener-specific patterns from a historical face video via cross-attention over non-adjacent segments and adds them to sampled rea
Key results
- PerReactor reaches FRCorr 0.34 (MAFRG) and 0.29 (PMAFRG) on REACT2024 test, versus 0.19 and 0.15-0.17 for the best prior methods (REGNN, Unifarn), roughly 71-79% gains.
- Ablation: appropriateness discriminator DG_A raises FRCorr from 0.06 to 0.24 for MAFRG, and the full model reaches 0.34.
- Using both audio and visual speaker behaviour gives the highest FRCorr (0.31 MAFRG, 0.26 PMAFRG) versus unimodal inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace reconstruction-loss training with the appropriateness discriminator DG_A for GAG (ablation) | PerReactor GAG · REACT2024 test set, MAFRG | FRCorr: 0.06 to 0.24 | Helped |
| Replace reconstruction-loss training with the appropriateness discriminator DP_A for PAG (ablation) | PerReactor PAG · REACT2024 test set, PMAFRG | FRCorr: 0.02 to 0.29 | Helped |
| Add realism discriminator DR to GAG (ablation) | PerReactor GAG · REACT2024 test set, MAFRG | FRCorr: 0.06 to 0.24 | Helped |
| Add PAG to GAG without appropriateness discriminators (ablation) | PerReactor · REACT2024 test set, MAFRG | FRCorr: 0.06 to 0.02 | Hurt |
| Add PAG to GAG + DR + DG_A (ablation) | PerReactor · REACT2024 test set, MAFRG | FRCorr: 0.31 to 0.30 | Hurt |
| Use both speaker audio and visual behaviour instead of visual behaviour alone (ablation) | PerReactor · REACT2024 test set, MAFRG | FRCorr: 0.26 to 0.31 | Helped |
| Use both speaker audio and visual behaviour instead of visual behaviour alone (ablation) | PerReactor · REACT2024 test set, PMAFRG | FRCorr: 0.23 to 0.26 | Helped |
| Use both speaker audio and visual behaviour instead of visual behaviour alone (ablation) | PerReactor · REACT2024 test set | FRDiv: 15.17 to 17.87 | Helped |
| Use 28.8s rather than 19.2s of historical face video (ablation) | PerReactor personalised behaviour pattern modelling · REACT2024 test set, PMAFRG | FRCorr: 0.29 to 0.30 | Helped |
| Use different historical video contents (ablation) | PerReactor personalised behaviour pattern modelling · REACT2024 test set, PMAFRG | FRCorr: 0.29 to 0.30 | No clear effect |
Surprising findings
- Using a realism discriminator alone without appropriateness discriminators yields very poor appropriateness (FRCorr about 0.02-0.03).
- FRdist is worse than some baselines, as higher diversity comes at the cost of larger distances to real reactions.
Limitations
- Relatively limited FRDist performance, as acknowledged by the authors.
- Visualisation performance is limited.
- Evaluated only on the REACT2024 MAFRG dataset (NoXI and RECOLA), which is small (1594 training pairs).
Experimental setup
Datasets: REACT2024 MAFRG dataset (NoXI, RECOLA)
Benchmarks: REACT2024 test set
Metrics: FRCorr FRdist FRDiv FRDvs FRVar FRRea FRSyn
Baselines: Trans-VAE REGNN Unifarn Beamer FRDiff BeLFusion USTC-AC
Builds on: Trans-VAE conditional GAN REACT2024 MAFRG benchmark
Closest papers by meaning
- FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · similarity 0.85)
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.85)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.85)
- GLARE: Generating Listening Heads with Appropriate REactions (NeurIPS 2026 · similarity 0.84)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.84)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.