Research map / Label-efficient and robust learning / Recognition and long-tail
Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NFER-Former, a ViT-based transformer with endogenously orthogonal self-attention decomposition (SAOD) and a hypergraph-guided AU-expression feature embedding (HGFE) for near-infrared facial expression recognition, plus a new 360-subject NIR-VIS dataset Large-HFE.
Problem
Facial expression recognition from near-infrared (NIR) images is hard due to scarce NIR training data, large NIR-VIS modality gap (intra-class variation), and high inter-class similarity among expressions.
Approach
Use a dual-head self-attention whose value projection is parameterized by Householder-based orthogonal matrices so spectrum and expression subspaces are orthogonal by construction (no extra regularizer), add a spectrum token supervised for spectrum classification, and model AU-expression co-occurrence as a hypergraph processed by HGNN to produce attention-weighted expression features, leveraging l
Key results
- NFER-Former reaches 84.03% accuracy and 83.82% F1 on Oulu-CASIA under NIR+VIS+Extra VIS, versus 80.53% for the best FER baseline (SCN).
- On Large-HFE under the same setting, NFER-Former achieves 82.17% accuracy and 81.99% F1, above the best baseline DFD at 79.83%.
- Ablation: baseline ViT 81.21% on Oulu-CASIA; SAOD alone 82.71%, HGFE alone 82.15%, both together 84.03%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NFER-Former instead of the strongest baseline SCN | NIR FER · Oulu-CASIA, NIR + VIS | accuracy: 80.04±0.95 -> 81.95±0.60 | Helped |
| Use NFER-Former instead of the strongest baseline SCN | NIR FER · Large-HFE, NIR + VIS | accuracy: 80.04±0.68 -> 80.49±0.90 | Helped |
| Use NFER-Former instead of the strongest baseline | NIR FER · Oulu-CASIA, NIR + VIS + Extra VIS | accuracy: 80.53±0.71 -> 84.03±0.65 | Helped |
| Use NFER-Former instead of the strongest baseline | NIR FER · Large-HFE, NIR + VIS + Extra VIS | accuracy: 79.83±1.04 -> 82.17±1.08 | Helped |
| Add SAOD to the Vision Transformer baseline (ablation) | NFER-Former · Oulu-CASIA | accuracy: 81.21 -> 82.71 | Helped |
| Add SAOD to the Vision Transformer baseline (ablation) | NFER-Former · Large-HFE | accuracy: 79.11 -> 80.51 | Helped |
| Add HGFE to the Vision Transformer baseline (ablation) | NFER-Former · Oulu-CASIA | accuracy: 81.21 -> 82.15 | Helped |
| Add HGFE to the Vision Transformer baseline (ablation) | NFER-Former · Large-HFE | accuracy: 79.11 -> 80.78 | Helped |
| Add both SAOD and HGFE to the Vision Transformer baseline (ablation) | NFER-Former · Oulu-CASIA | accuracy: 81.21 -> 84.03 | Helped |
| Add both SAOD and HGFE to the Vision Transformer baseline (ablation) | NFER-Former · Large-HFE | accuracy: 79.11 -> 82.17 | Helped |
| Vary the spectrum-loss balance parameter λ; the paper selects λ=0.1 (ablation) | NFER-Former · Oulu-CASIA and Large-HFE | recognition performance | Helped |
Surprising findings
- The HFR method DFD scores only 64.58% on Oulu-CASIA under NIR+VIS, below NIR-only training for several FER baselines.
- Adding VIS data does not uniformly help: several baselines (e.g., MA-Net, DFD on Oulu-CASIA) drop when VIS samples are added.
Limitations
- Evaluated only on two NIR FER datasets, one of which is newly collected by the authors.
- Relies on lab-controlled image collection with a limited number of expression classes (six basic plus neutral).
Experimental setup
Datasets: Oulu-CASIA Large-HFE KDEF CFEE
Benchmarks: Oulu-CASIA Large-HFE
Metrics: accuracy macro F1
Baselines: ResNet IL-CNN SCN MA-Net FDRL DSVNs DFD HiFaceGAN T2V-DDPM
Builds on: Vision Transformer HGNN Householder transformation
Closest papers by meaning
- Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition (AAAI 2024 · similarity 0.92)
- Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-Spectral Hallucination and Low-Rank Embedding (CVPR 2017 · similarity 0.86)
- Open-Set Facial Expression Recognition (AAAI 2024 · similarity 0.85)
- Navigating Label Ambiguity for Facial Expression Recognition in the Wild (AAAI 2025 · similarity 0.84)
- Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild (AAAI 2025 · similarity 0.84)
- FAIR-FER: A Latent Alignment Approach for Mitigating Bias in Facial Expression Recognition (Student Abstract) (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.