Research map / Video / Video segmentation and tracking
Multi-Modal Hand-to-Mouth Gesture Recognition in Activity-Oriented RGB-Thermal Footage (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds RGB-Thermal fusion models (early, decision, and a new mid-fusion RTFVNet) on TimeSformer and a ResNet50-based encoder to classify eating, smoking, and non-gesture activity in neck-worn camera footage.
Problem
Distinguishing confounding hand-to-mouth gestures (eating vs. smoking vs. none) from wearable activity-oriented footage, where RGB and thermal single modalities each have sensing limitations.
Approach
RTFVNet is a mid-fusion architecture adapted from RTFNet that removes cross-modal shortcut connections, uses a self-attention module to compute per-modality weights applied to ResNet50 embeddings, and feeds fused features into an LSTM for temporal classification.
Key results
- Decision fusion achieved the highest F1-score of 87.7% at 0.44 TFLOPs.
- RTFVNet achieved F1-score 84.8% at 0.37 TFLOPs, versus 82.6% for RGB-only and 79.6% for thermal-only.
- Early fusion reached F1-score 83.8% at 0.22 TFLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse outputs from separate RGB and thermal models instead of using RGB alone | TimeSformer · Eating, smoking, and non-gesture activities | F1-score: 82.6 ± 2.4 -> 87.7 ± 1.6 | Helped |
| Fuse outputs from separate RGB and thermal models instead of using thermal alone | TimeSformer · Eating, smoking, and non-gesture activities | F1-score: 79.6 ± 2.0 -> 87.7 ± 1.6 | Helped |
| Use RTFVNet mid-fusion instead of RGB-only input | HTM gesture classification · Eating, smoking, and non-gesture activities | F1-score: 82.6 ± 2.4 -> 84.8 ± 2.2 | Helped |
| Use RTFVNet mid-fusion instead of thermal-only input | HTM gesture classification · Eating, smoking, and non-gesture activities | F1-score: 79.6 ± 2.0 -> 84.8 ± 2.2 | Helped |
| Use decision fusion instead of RTFVNet mid-fusion | RGB-Thermal gesture recognition · Eating, smoking, and non-gesture activities | F1-score: 84.8 ± 2.2 -> 87.7 ± 1.6 | Helped |
| Use RTFVNet mid-fusion instead of decision fusion | RGB-Thermal gesture recognition · Eating, smoking, and non-gesture activities | TFLOPs: 0.44 -> 0.37 | Helped |
| Use early fusion instead of RGB-only input | TimeSformer · Eating, smoking, and non-gesture activities | F1-score: 82.6 ± 2.4 -> 83.8 ± 2.2 | Helped |
| Use early fusion instead of thermal-only input | TimeSformer · Eating, smoking, and non-gesture activities | F1-score: 79.6 ± 2.0 -> 83.8 ± 2.2 | Helped |
| Use RTFVNet mid-fusion instead of early fusion | RGB-Thermal gesture recognition · Eating, smoking, and non-gesture activities | F1-score: 83.8 ± 2.2 -> 84.8 ± 2.2 | Helped |
Limitations
- Thermal sensors are affected by ambient temperature and occlusions, as noted by the authors.
- Dataset limited to 7 smokers.
- Decision fusion is computationally costly for real-time pipelines.
Experimental setup
Datasets: HabitSense RGB-Thermal footage (441.9 hours, 7 smokers)
Benchmarks: HabitSense RGB-Thermal dataset
Metrics: F1-score precision recall TFLOPs
Baselines: RGB-only TimeSformer thermal-only TimeSformer decision fusion early fusion
Builds on: TimeSformer RTFNet
Closest papers by meaning
- DarkAct: A RGB-Thermal Dataset and Fusion Framework for Multimodal Low-Light Action Recognition (CVPR 2026 · similarity 0.84)
- ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment (NeurIPS 2022 · similarity 0.83)
- RhythmMamba: Fast, Lightweight, and Accurate Remote Physiological Measurement (AAAI 2025 · similarity 0.82)
- Fusion-Vital: Video-RF Fusion Transformer for Advanced Remote Physiological Measurement (AAAI 2024 · similarity 0.82)
- X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing (ICLR 2025 · similarity 0.81)
- Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.