Research map / Vision-language and multimodal / Multimodal LLMs

VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VowelPrompt, which converts forced-aligned vowel-level prosodic descriptors (F0, intensity, duration) into natural-language text appended to transcripts, then adapts LLMs with SFT followed by GRPO for speech emotion recognition.

Problem

Text-only LLM emotion recognition from speech transcripts ignores fine-grained prosody, and utterance-level prosodic descriptions are coarse; audio-embedding LLMs are opaque.

Approach

Extract vowel-segment LLDs (pitch level/slope/variation, intensity level/variation, duration) via forced alignment, normalize per speaker and vowel type, quantile-bin into words, and append to the transcript; adapt the LLM with SFT on GPT-4o reasoning traces and GRPO with accuracy and format rewards.

Key results

• Zero-shot with GPT-4o on IEMOCAP (transcript and context) reaches 62.26 UACC / 60.74 WF1, versus 55.51 / 53.63 for the zero-shot baseline.
• With LLaMA-3-8B SFT and GRPO, WF1 reaches 73.02 on IEMOCAP and 68.98 on MELD.
• Cross-domain MELD to IEMOCAP with SFT and GRPO reaches 51.75 vs 44.79 for SpeechCueLLM.

Surprising findings

• Prosody permutation drops MELD WF1 to 41.72, while transcript shuffle only drops it to 67.00, suggesting the model relies on aligned prosody rather than word order.
• Adding consonant-level descriptors does not beat vowel-only on French or English, though it helps slightly on German.

Limitations

• Relies on GPT-4o-generated reasoning traces for SFT and on a 20% training subset for SFT and GRPO.
• Many baseline numbers and some tables appear internally inconsistent (e.g., the same VowelPrompt value reported under different settings), which weakens confidence in the reported results.
• Zero-shot and cross-domain gains over baselines are modest in several settings.

Experimental setup

Datasets: IEMOCAP MELD CaFE EmoDB ASVP-ESD

Benchmarks: IEMOCAP MELD CaFE EmoDB ASVP-ESD

Metrics: weighted F1 unweighted accuracy

Baselines: Zero-Shot Baseline SpeechCueLLM InstructERC SALMONN HuBERT-large wav2vec-large

Builds on: SpeechCueLLM GRPO RLVR openSMILE Montreal Forced Aligner

Closest papers by meaning

• EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning (ICLR 2026 · similarity 0.90)
• ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis (AAAI 2025 · similarity 0.87)
• EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.86)
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.86)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.85)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.85)