Research map / Vision-language and multimodal / Cross-modal learning

Mental-Perceiver: Audio-Textual Multi-Modal Learning for Estimating Mental Disorders

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMPsy, a large Mandarin adolescent audio-text corpus for anxiety/depression assessment, and Mental-Perceiver, a Perceiver-IO-style attention network with category semantic priors and a matching loss for detecting these disorders from audio and transcripts.

Problem

Automatic detection of anxiety and depression from speech and text is limited by the lack of large, publicly available datasets, especially for adolescents and non-English languages.

Approach

A fully attentional Perceiver-style network whose latent array is initialized from class-wise semantic priors (centroids of text representations for normal and disorder classes), trained with an added matching loss, fusing audio and transcript features for binary mental disorder estimation.

Key results

• On MMPsy-Anxiety, Mental-Perceiver reaches 0.85 accuracy and 0.76 UAR, versus 0.88 accuracy for XGBoost (higher accuracy but lower UAR of 0.60).
• On MMPsy-Depression, Mental-Perceiver achieves 0.85 accuracy, 0.79 UAR and 0.69 sensitivity, outperforming PerceiverIO (0.81 accuracy, 0.77 UAR).
• On DAIC-WOZ, Mental-Perceiver achieves 0.79 accuracy and 0.66 UAR, with specificity 0.97.

Measured effects reported in the paper

Surprising findings

• Audio-only input on MMPsy-Anxiety yields high accuracy (0.87) but very low sensitivity (0.10), showing accuracy is misleading under class imbalance.
• The zero-shot-style Qwen2-Audio-Instruct baseline is competitive on sensitivity but has low precision on the positive class.

Limitations

• Strong class imbalance in MMPsy-Anxiety; model accuracy gains do not uniformly translate into better UAR or sensitivity on the positive class.
• Data limited to Mandarin-speaking adolescents from one province; generalizability is unclear.
• Binary labels derived from self-reported GAD-7/PHQ-9 questionnaires rather than clinical diagnosis.

Experimental setup

Datasets: MMPsy DAIC-WOZ

Benchmarks: MMPsy-Anxiety MMPsy-Depression DAIC-WOZ

Metrics: Accuracy UAR Sensitivity Specificity Precision Recall F1

Baselines: SVM RandomForest XGBoost NUSD ConvLSTM PerceiverIO AFABNet Qwen2-Audio-Instruct

Builds on: PerceiverIO Transformer attention BERT AdamW

Closest papers by meaning

• Thousand Voices of Trauma: A Large-Scale Synthetic Dataset for Modeling Prolonged Exposure Therapy Conversations (NeurIPS 2025 · similarity 0.85)
• EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models (ICLR 2026 · similarity 0.84)
• MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis (NeurIPS 2025 · similarity 0.84)
• MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents (AAAI 2025 · similarity 0.82)
• From Crayons to Code: AI-Driven Insights into a Child’s Mental Health Through Drawings (AAAI 2025 · similarity 0.82)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.82)