Research map / Label-efficient and robust learning / Recognition and long-tail
AFFAKT: A Hierarchical Optimal Transport Based Method for Affective Facial Knowledge Transfer in Video Deception Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFFAKT, a hierarchical optimal transport method that transfers facial-expression knowledge from large VFER datasets to improve video deception detection on RLT and DOLOS.
Problem
Video deception detection suffers from scarce high-quality labeled data; the paper asks how much facial-expression knowledge to transfer from a large VFER dataset and how to use it at inference.
Approach
Quantifies class-to-sample correlation via hierarchical optimal transport (H-OTKT) between VFER classes and deception samples, and keeps a momentum-updated correlation prototype (SRKB) that re-weights each test sample's transport plan based on its transport-plan standard deviation.
Key results
- On RLT with visual modality and DFEW source, AFFAKT reaches F1 0.8760, ACC 0.8670, AUC 0.8789, versus 0.7793 F1 for ADC.
- With fused modalities, AFFAKT with MAFW source reaches ACC 0.8427 and AUC 0.8563 on RLT.
- Learned correlation prototype associates deception with 'sad' and truthfulness with 'happy', consistent with psychology findings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer facial-expression knowledge using AFFAKT rather than the strongest listed baseline | video deception detection · RLT visual; DFEW source | ACC: 0.8173 -> 0.8670 | Helped |
| add H-OTKT knowledge transfer (ablation) | encoder-and-classifier baseline · RLT visual; DFEW source | ACC: 0.7840 -> 0.8093 | Helped |
| add H-OTKT knowledge transfer (ablation) | encoder-and-classifier baseline · DOLOS visual; DFEW source | ACC: 0.6647 -> 0.6677 | Helped |
| add H-OTKT knowledge transfer (ablation) | encoder-and-classifier baseline · RLT audio and fused; FERV39K or MAFW sources | ACC | Mixed |
| add SRKB to H-OTKT, using fixed σ_i=0.2 (ablation) | AFFAKT · RLT and DOLOS, audio and fused modalities | ACC | Mixed |
| use proposed sample-specific re-weighting instead of fixed σ_i=0.2 (ablation) | SRKB · DOLOS audio and fused modalities | ACC | Helped |
| replace MAE-DFER with Former-DFER for source feature extraction (ablation) | AFFAKT · RLT and DOLOS | ACC | Hurt |
Surprising findings
- Pre-trained source encoder choice strongly affects results; different VFER source datasets give best performance for different target datasets and modalities.
- H-OTKT alone lacks robustness: ACC decreases on some target datasets and modalities relative to the baseline.
Limitations
- Results depend on the quality of the pre-trained VFER encoder and source dataset choice.
- Small target datasets (121 and 1675 videos) and 5-fold evaluation yield high variance across folds.
Experimental setup
Datasets: RLT DOLOS DFEW FERV39K MAFW
Benchmarks: RLT DOLOS
Metrics: F1 score ACC AUC
Baselines: OpenFace + SVM AU + LSTM ResNet18 + LSTM PECL FreeLunch ADC Cr-KD-NCD W2V2 + MLP
Builds on: FreeLunch ADC PECL Sinkhorn algorithm UT-Adapter
Closest papers by meaning
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.83)
- Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild (AAAI 2025 · similarity 0.83)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.83)
- Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (CVPR 2025 · similarity 0.82)
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models (NeurIPS 2025 · similarity 0.82)
- DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.