Research map / Vision-language and multimodal / Cross-modal learning
Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 287-participant full-body pose multimodal personality dataset and proposes PINet (Vision Mamba encoder, Fusion Mamba interaction, psychology-informed modality correlation loss) for Big Five trait regression.
Problem
Predicting Big Five personality traits from multimodal data performs poorly, and existing datasets and models lack frontal full-body pose data and ignore per-dimension modality relevance.
Approach
Treats full-body pose as a modality, encodes video with Vision Mamba and fuses modalities with Fusion Mamba, and adds a loss that weights specific modalities for specific Big Five dimensions based on psychological hypotheses.
Key results
- PINet achieves average PCC 0.2509 and CCC 0.2706, versus 0.1141 and 0.1124 for the second-best CRNet.
- PINet attains the best MSE (0.0277) and ACC (0.8679) among all compared models on the averaged results.
- Adding pose data raises PCC by 36.8% for PINet, 45.9% for CRNet, and 25.5% for ResNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PINet instead of CRNet | Personality recognition on our dataset · Our dataset (average across five traits) | PCC: 0.1141 -> 0.2509 | Helped |
| Use PINet instead of CRNet | Personality recognition on our dataset · Our dataset (average across five traits) | CCC: 0.1124 -> 0.2706 | Helped |
| Use PINet instead of GatedGCN | Personality recognition on our dataset · Our dataset (average across five traits) | ACC: 0.8651 -> 0.8679 | Helped |
| Use PINet instead of EmoFormer | Personality recognition on our dataset · Our dataset (average across five traits) | MSE: 0.0293 -> 0.0277 | Helped |
| Add the Vision Mamba Block instead of a Vision Transformer (ablation) | PINet · Our dataset (average across five traits) | PCC | Helped |
| Add MFI instead of simple concatenation (ablation) | PINet · Our dataset (average across five traits) | PCC | Helped |
| Add PIMC Loss (ablation) | PINet · Our dataset (average across five traits) | PCC | Helped |
| Add pose data (ablation) | PINet · Our dataset (average across five traits) | PCC: +36.8% | Helped |
| Add pose data (ablation) | CRNet · Our dataset (average across five traits) | PCC: +45.9% | Helped |
| Add pose data (ablation) | ResNet · Our dataset (average across five traits) | PCC: +25.5% | Helped |
| Add pose data (ablation) | PINet · Our dataset (average across five traits) | CCC: +172.5% | Helped |
| Add pose data (ablation) | ResNet · Our dataset (average across five traits) | CCC: +26.3% | Helped |
Surprising findings
- The SIAP study reported pose as the least effective modality, whereas this paper finds pose has moderate importance across modalities.
- Adding pose data did not improve CCC for CRNet, despite improving the other metrics.
Limitations
- Labels are self-reported Big Five scores only; no observer-reported labels were used.
- Absolute performance remains modest (average PCC around 0.25), and the dataset is limited to 287 Chinese university students.
- PIMC Loss modality assignments are based on hypotheses from psychological theory rather than empirically validated mappings.
Experimental setup
Datasets: open-source audio-visual benchmark PINet personality dataset (287 participants)
Benchmarks: self-reported Big Five personality dataset (proposed) open-source audio-visual benchmark
Metrics: PCC CCC ACC MSE
Baselines: PersEmoN SENet GatedGCN HRNet EmoFormer ResNet CRNet
Builds on: Mamba Video Mamba Fusion Mamba AlphaPose CRNet
Closest papers by meaning
- Multimodal Fine-Grained Apparent Personality Trait Recognition: Joint Modeling of Big Five and Questionnaire Item-level Scores (AAAI 2025 · similarity 0.89)
- PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits (ICLR 2026 · similarity 0.88)
- egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-world Tasks (NeurIPS 2025 · similarity 0.86)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.84)
- Personality Alignment of Large Language Models (ICLR 2025 · similarity 0.83)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.