Research map / Image generation / Editing and style transfer
Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NFER-SDE, a score-based diffusion model that translates visible-light facial images to near-infrared using landmark and high-pass frequency energy guidance plus conditional score matching, to augment small NIR facial expression datasets.
Problem
Near-infrared facial expression recognition suffers from very little NIR training data and a large modality gap from visible-light images; existing heterogeneous image synthesis lacks controllability for FER.
Approach
Guide the reverse SDE of a score-based diffusion model with two task-specific energies (facial landmark heatmap consistency and high-pass frequency consistency, motivated by high-frequency content carrying modality-invariant expression cues) and condition the score network on the VIS source image by channel concatenation.
Key results
- On Oulu-CASIA, NFER-SDE reaches 70.19% accuracy and 70.28% F1 versus 66.94% for the best baseline (HiFaceGAN), with LPIPS 0.1827 and PSNR 65.75.
- On Large-HFE, NFER-SDE reaches 74.90% accuracy and 74.91% F1, versus 72.59% for T2V-DDPM.
- Ablation on Oulu-CASIA shows the baseline SBDM at 62.55% accuracy rising to 70.19% with conditional score matching plus both energy functions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NFER-SDE instead of the strongest baseline | NIR FER · Oulu-CASIA | accuracy: 3.25% | Helped |
| Add NFER-SDE instead of the strongest baseline | NIR FER · Large-HFE | accuracy: 2.31% | Helped |
| Use conditional score-matching instead of unconditional score-matching (ablation) | NFER-SDE · Oulu-CASIA | accuracy: 62.55 -> 65.76 | Helped |
| Use conditional score-matching instead of unconditional score-matching (ablation) | NFER-SDE · Large-HFE | accuracy: 69.12 -> 71.99 | Helped |
| Add facial-landmark energy E_h to conditional score-matching (ablation) | NFER-SDE · Oulu-CASIA | accuracy: 65.76 -> 67.78 | Helped |
| Add facial-landmark energy E_h to conditional score-matching (ablation) | NFER-SDE · Large-HFE | accuracy: 71.99 -> 72.39 | Helped |
| Add high-frequency energy E_f to conditional score-matching (ablation) | NFER-SDE · Oulu-CASIA | accuracy: 65.76 -> 68.24 | Helped |
| Add high-frequency energy E_f to conditional score-matching (ablation) | NFER-SDE · Large-HFE | accuracy: 71.99 -> 73.59 | Helped |
| Combine conditional score-matching with both task-specific energy functions (ablation) | NFER-SDE · Oulu-CASIA | accuracy: 62.55 -> 70.19 | Helped |
| Combine conditional score-matching with both task-specific energy functions (ablation) | NFER-SDE · Large-HFE | accuracy: 69.12 -> 74.90 | Helped |
Surprising findings
- Plain diffusion baselines (ILVR, EGSDE) fall below the no-translation accuracy on Oulu-CASIA, indicating that generic image-translation diffusion can hurt FER.
Limitations
- Evaluated only on two small lab-controlled NIR FER datasets.
- Reported improvements rely on the translated CFEE/KDEF VIS data and per-experiment random splits averaged over five runs.
Experimental setup
Datasets: Oulu-CASIA Large-HFE CFEE KDEF
Benchmarks: Oulu-CASIA Large-HFE
Metrics: accuracy macro F1 LPIPS PSNR SSIM
Baselines: pix2pix CycleGAN HiFaceGAN ILVR SR3 SDEdit EGSDE T2V-DDPM
Builds on: EGSDE SDEdit score-based diffusion models ILVR T2V-DDPM
Closest papers by meaning
- Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition (AAAI 2024 · similarity 0.92)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.84)
- Navigating Label Ambiguity for Facial Expression Recognition in the Wild (AAAI 2025 · similarity 0.84)
- Open-Set Facial Expression Recognition (AAAI 2024 · similarity 0.84)
- FAIR-FER: A Latent Alignment Approach for Mitigating Bias in Facial Expression Recognition (Student Abstract) (AAAI 2024 · similarity 0.83)
- Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.