Research map / Vision-language and multimodal / Human and embodied modeling
Multimodal Fine-Grained Apparent Personality Trait Recognition: Joint Modeling of Big Five and Questionnaire Item-level Scores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-task and cascaded joint models that predict Big Five scores and 50 questionnaire item-level scores from audio-visual-text self-introduction videos using a multimodal transformer, and releases a new 10,100-video dataset with 50-item Big Five annotations.
Problem
Apparent personality trait recognition from multimodal behavior usually regresses only Big Five scores, discarding the fine-grained item-level answers in questionnaire-based personality tests; existing datasets also lack such item-level results.
Approach
Jointly model Big Five scores and questionnaire item-level scores, either via a multi-task model with two heads on a shared multimodal transformer or a cascaded model that predicts item scores first and then Big Five scores, trained with MAE on both targets.
Key results
- The multi-task joint model with audio, text, and visual inputs reached average Big Five correlation 0.677 and accuracy 93.8, versus 0.643 / 93.5 for the single-task model.
- Multi-task joint modeling for questionnaire item-level scores achieved average correlation 0.557 versus 0.546 for the single-task model (accuracy 90.8 for both).
- Best model's Big Five estimation matched human evaluation (average correlation 0.634 and accuracy 92.4 for humans) on the test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly estimate Big Five and questionnaire item-level scores with a multi-task model instead of estimating Big Five alone | Multimodal transformer · Big Five test | Pearson correlation (average): 0.643 -> 0.677 | Helped |
| Use cascaded joint modeling instead of single-task estimation | Multimodal transformer · Big Five test | Pearson correlation (average): 0.643 -> 0.643 | No clear effect |
| Jointly estimate Big Five and questionnaire item-level scores with a multi-task model instead of estimating item scores alone | Multimodal transformer · Questionnaire item-level scores | Pearson correlation (average): 0.546 -> 0.557 | Helped |
| Use cascaded joint modeling instead of estimating questionnaire item-level scores alone | Multimodal transformer · Questionnaire item-level scores | accuracy (average): 90.8 -> 90.7 | Hurt |
| Increase the number of jointly modeled questionnaire items from 0 to 50 (ablation) | Multi-task joint model · Big Five test | Pearson correlation (average): 0.643 -> 0.677 | Helped |
| Add audio, text, and visual inputs instead of audio input alone (ablation) | Single-task model · Big Five test | Pearson correlation (average): 0.332 -> 0.471 | Helped |
| Pre-train the audio encoder (ablation) | Single-task model with audio input · Big Five test | Pearson correlation (average): 0.332 -> 0.561 | Helped |
| Pre-train the visual encoder (ablation) | Single-task model with visual input · Big Five test | Pearson correlation (average): 0.145 -> 0.294 | Helped |
| Pre-train both audio and visual encoders (ablation) | Single-task model with audio and visual input · Big Five test | Pearson correlation (average): 0.251 -> 0.606 | Helped |
| Pre-train audio, text, and visual encoders (ablation) | Single-task model with audio, text, and visual input · Big Five test | Pearson correlation (average): 0.471 -> 0.643 | Helped |
Surprising findings
- The cascaded joint model performed comparably to the single-task model, unlike the multi-task variant, suggesting direct auxiliary supervision works better than cascading.
- Model item-level estimates outperformed human evaluation (average correlation 0.438 for humans), which the authors attribute to more stable multimodal criteria.
Limitations
- Dataset is limited to Japanese participants with self-introduction videos recorded in a fixed frontal setting.
- Joint modeling gains for questionnaire item-level estimation were small (multi-task only slightly better than single-task).
Experimental setup
Datasets: ChaLearn first impression UDIVA VGGFace2 FER RAF-DB AffectNet Wider Face
Benchmarks: self-introduction video dataset with 50-item Big Five results
Metrics: Pearson correlation coefficient accuracy
Baselines: single-task model cascaded joint model human evaluation
Builds on: multimodal transformer MobileNetV3 masked language modeling (BERT) HuBERT-style masked prediction
Closest papers by meaning
- Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset (AAAI 2025 · similarity 0.89)
- PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits (ICLR 2026 · similarity 0.87)
- Idiographic Personality Gaussian Process for Psychological Assessment (NeurIPS 2024 · similarity 0.85)
- Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts (NeurIPS 2026 · similarity 0.84)
- Neuron based Personality Trait Induction in Large Language Models (ICLR 2025 · similarity 0.83)
- LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.