Research map / Label-efficient and robust learning / Recognition and long-tail
T-FAKE: Synthesizing Thermal Images for Facial Landmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the T-FAKE synthetic thermal face dataset (200k frames, 70 sparse and 478 dense landmarks) by thermalizing the FAKE RGB dataset with a semi-supervised RGB2Thermal loss, and trains thermal/RGB+thermal landmark predictors on it.
Problem
Thermal facial landmark detection suffers from a lack of large-scale annotated thermal face datasets, and no dense thermal landmarker existed.
Approach
A semi-supervised RGB2Thermal U-Net loss combining paired supervised reconstruction, a Wasserstein patch-distribution style term, and a segmentation-based clinical temperature prior, used to thermalize synthetic RGB faces into a large thermal landmark dataset.
Key results
- On CHARLOTTE, the T-FAKE sparse landmarker with a confidence threshold of 6e-4 reaches full NME W/H 0.0889 and 5.79% failure rate, the best sparse thermal result among compared methods.
- The RGB2Thermal model with both regularizers (λR=1, λW=0.01C) gives FID 0.1598 and TUFTS MSE 0.0353, outperforming Pix2Pix, CycleGAN, CUT, and QS-Attn on temperature prediction.
- Training with FAKE+T-FAKE improves 300W NME IO from 0.0570 (FAKE only) to 0.0522 without degrading RGB performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add T-FAKE thermal training data to FAKE | GLL+SW Sparse · CHARLOTTE Full | NME W/H: 0.1887 -> 0.0969 | Helped |
| tighten the face-detection threshold from ¯σ < 6 × 10−4 to ¯σ < 2.2 × 10−4 | GLL+SW Sparse · CHARLOTTE Full | NME W/H: 0.0969 -> 0.0815 | Helped |
| tighten the face-detection threshold from ¯σ < 6 × 10−4 to ¯σ < 2.2 × 10−4 | GLL+SW Sparse · CHARLOTTE Full | failure rate (%): 5.79 -> 20.00 | Hurt |
| add T-FAKE training data to FAKE | GLL+SW · 300W test | NME IO: 0.0570 -> 0.0522 | Helped |
| add segmentation-based and patch-based regularizers (ablation) | RGB2Thermal · Thermalized FAKE vs SEJONG | FID: .5028 -> .1598 | Helped |
| add segmentation-based regularizer (ablation) | RGB2Thermal · Thermalized FAKE vs SEJONG | FID: .3146 -> .1598 | Helped |
| add patch-based regularizer (ablation) | RGB2Thermal · Thermalized FAKE vs SEJONG | FID: .1706 -> .1598 | Helped |
| add segmentation-based and patch-based regularizers (ablation) | RGB2Thermal · TUFTS | MSE: .0378 -> .0353 | Helped |
| add segmentation-based and patch-based regularizers (ablation) | RGB2Thermal · DRIVE-IN | MSE: .1117 -> .0580 | Helped |
Surprising findings
- QS-Attn achieves the best FID among compared thermalization models yet fails at temperature prediction, especially on non-frontal DRIVE-IN images.
- Mediapipe with a pre-processing stack reduces failure rates drastically for RGB landmarkers on thermal images.
Limitations
- Dense ground truth is pseudo-labeled by MediaPipe predictions rather than manual annotation.
- Thermalization is trained on lab-recorded SEJONG data of 100 subjects with frontal views and controlled lighting; generalization to in-the-wild is evaluated only indirectly.
- Temperature regularizer relies on hardcoded reference temperatures from empirical studies, with only cold and warm settings.
Experimental setup
Datasets: SEJONG FAKE CHARLOTTE 300W TUFTS DRIVE-IN MediaPipe-derived dense annotations
Benchmarks: CHARLOTTE 300W TUFTS DRIVE-IN
Metrics: NME failure rate FID MSE
Baselines: Pix2Pix CycleGAN CUT QS-Attn Star 3FabRec Mediapipe FA DAN YOLO5Face Dlib U-Net+Wing
Builds on: FAKE synthetic face dataset Pix2Pix entropy-regularized Wasserstein distance MobileNetV2 Gaussian log-likelihood landmark regression (LUVLi-style)
Closest papers by meaning
- 3D-Aware Facial Landmark Detection via Multi-View Consistent Training on Synthetic Data (CVPR 2023 · similarity 0.85)
- Generalizable Face Landmarking Guided by Conditional Face Warping (CVPR 2024 · similarity 0.84)
- Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion (ICML 2025 · similarity 0.84)
- ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translation (NeurIPS 2025 · similarity 0.84)
- TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction (ICLR 2025 · similarity 0.83)
- POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Search (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.