Research map / Vision-language and multimodal / Cross-modal learning
MMPF: Multi-Modal Perception Framework for Abnormal Medical Condition Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMPF, a multi-stream image-based framework fusing whole image, facial expression, 2D skeleton and 3D pose features via a Cascade Squeeze-Excitation module to detect abnormal medical conditions, and releases a new 10,500-image dataset.
Problem
Non-invasive detection of abnormal medical conditions (e.g., heart attack, headache, stomachache, falls) from images, where prior methods ignore facial expressions and body posture or focus on a single condition.
Approach
Combines facial-expression and body-posture (2D skeletal map and lifted 3D coordinates) streams with whole-image features, using a Cascade Squeeze-Excitation module (Adaptive SE per modality plus Multi-modal SE for cross-modal fusion) and random joint occlusion augmentation.
Key results
- Best configuration (EfficientNet whole-image + 3DCNN streams with CSE and RJO) reaches 88.13% accuracy and 87.74 F1 on the new dataset.
- On NTU RGB+D medical subset, the proposed method reaches 82.51% accuracy, above the best baseline Gul et al. 2020 at 77.18%.
- Ablation: MSE plus ASE within CSE yields 88.71% accuracy versus 86.73% for simple concatenation on MobileNet+3DCNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MMPF (ID4) instead of the strongest listed baseline | Abnormal medical condition detection · Our dataset | accuracy: 82.41 -> 88.13 | Helped |
| use MMPF (ID4) instead of the strongest listed baseline | Abnormal medical condition detection · NTU RGB+D | accuracy: 77.18 -> 82.51 | Helped |
| add face-image, 2D skeletal-map, and 3D-coordinate streams to the whole-image stream (ablation) | MMPF · Modality streams | Acc.: 79.83 -> 88.13 | Helped |
| add the face-image stream to the whole-image stream (ablation) | MMPF · Modality streams | Acc.: 79.83 -> 83.42 | Helped |
| add the 2D skeletal-map stream to whole image and face image (ablation) | MMPF · Modality streams | Acc.: 83.42 -> 86.77 | Helped |
| add 3D coordinates to the whole-image, face-image, and 2D skeletal-map streams (ablation) | MMPF · Modality streams | Acc.: 86.77 -> 88.13 | Helped |
| use only the face-image stream instead of all four modalities (ablation) | MMPF · Modality streams | Acc.: 88.13 -> 52.81 | Hurt |
| use only the 2D skeletal-map stream instead of all four modalities (ablation) | MMPF · Modality streams | Acc.: 88.13 -> 62.50 | Hurt |
| use only the 3D-coordinate stream instead of all four modalities (ablation) | MMPF · Modality streams | Acc.: 88.13 -> 63.15 | Hurt |
| combine ASE and MSE instead of using ASE with concatenation fusion (ablation) | MobileNet + 3DCNN · CSE module ablation | Acc.: 87.81 -> 88.71 | Helped |
| replace SE with ASE while keeping concatenation fusion (ablation) | MobileNet + 3DCNN · CSE module ablation | Acc.: 87.15 -> 87.81 | Helped |
| use MSE instead of concatenation fusion (ablation) | MobileNet + 3DCNN · CSE module ablation | Acc.: 86.73 -> 88.12 | Helped |
Surprising findings
- Deeper backbones give only marginal gains, suggesting the method is not heavily dependent on backbone choice.
- Body posture modalities carry more medical-condition information than facial expressions alone.
Limitations
- Dataset is small and partly reproduced by experimenters imitating symptoms, which may not reflect real facial expressions of patients.
- Evaluated on limited condition categories and not validated in clinical settings.
- Performance drops on NTU RGB+D due to multiple viewpoints and incomplete faces/postures.
Experimental setup
Datasets: NTU RGB+D Newly collected medical condition image dataset (10,500 images)
Benchmarks: Newly collected medical condition dataset NTU RGB+D (medical condition classes A43-A46)
Metrics: accuracy weighted F1 coarse-grained accuracy coarse-grained F1
Baselines: Rojas-Albarracin et al. 2019 Mohan et al. 2021 Khraief, Benzarti, and Amiri 2020 Gul et al. 2020
Builds on: SENet Lightweight OpenPose MT-CNN DeepFace lifting network
Closest papers by meaning
- Multi-modal Vision Pre-training for Medical Image Analysis (CVPR 2025 · similarity 0.84)
- Multi-modal Medical Diagnosis via Large-small Model Collaboration (CVPR 2025 · similarity 0.84)
- Beyond Single-Modal Boundary: Cross-Modal Anomaly Detection through Visual Prototype and Harmonization (CVPR 2025 · similarity 0.83)
- MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models (ICLR 2025 · similarity 0.82)
- MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment (AAAI 2024 · similarity 0.82)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.