Research map / Vision-language and multimodal / Human and embodied modeling
SECap: Speech Emotion Captioning with Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SECap, a HuBERT + Q-Former + LLaMA framework that generates natural-language speech emotion captions, trained with mutual-information disentanglement and contrastive learning on an internal Chinese dataset (EMOSpeech).
Problem
Speech emotion recognition reduces emotions to fixed discrete labels, which cannot capture complex or nuanced emotional expression; the paper targets describing speech emotions with natural language captions.
Approach
Introduces the speech emotion captioning (SEC) task and an encoder-bridge-decoder model where a Q-Former bridge is trained with speech-transcription mutual information minimization (vCLUB upper bound) to remove content information and speech-caption contrastive learning to capture emotion, with LLaMA as the caption decoder.
Key results
- SECap (Q-Emb only) reaches SIM1 71.95 vs 59.62 for HTSAT-BART baseline, outperforming it on all objective metrics.
- Subjective MOS of SECap is 3.77, above human labels (3.39) and on par with human caption annotations (3.85).
- Replacing BART with LLaMA and HTSAT with HuBERT plus Q-Former raises SIM1 from 59.62 to 71.95 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace HTSAT-BART with SECap | Speech emotion captioning · EMOSpeech test | SIM1: 59.62 -> 71.95 | Helped |
| replace HTSAT-BART with SECap | Speech emotion captioning · EMOSpeech test | BLEU1: 32.74 -> 36.08 | Helped |
| replace HTSAT-BART with SECap | Speech emotion captioning · EMOSpeech test | CIDEr: 2.21 -> 34.81 | Helped |
| use HuBERT instead of HTSAT (ablation) | Linear + BART captioning model · EMOSpeech test | SIM1: 7.26% | Helped |
| use LLaMA instead of BART (ablation) | HTSAT + Linear captioning model · EMOSpeech test | SIM1: 8.92% | Helped |
| replace the linear Bridge-Net with Q-Former (ablation) | HuBERT + LLaMA captioning model · EMOSpeech test | SIM1: 4.85% | Helped |
| use T-Embedding instead of raw transcription (ablation) | Text-only SECap · EMOSpeech test | SIM1: 56.49 -> 65.90 | Helped |
| use Q-Embedding alone instead of T-Embedding alone (ablation) | SECap · EMOSpeech test | SIM1: 9.18% | Helped |
| add raw transcription alongside Q-Embedding (ablation) | SECap · EMOSpeech test | SIM1: 3.77% | Hurt |
| add T-Embedding alongside Q-Embedding (ablation) | SECap · EMOSpeech test | SIM1: 69.66 vs. 71.95 | Hurt |
| add STMIL during stage 1 (ablation) | SECap Q-Former · EMOSpeech test | SIM1: 2.17% | Helped |
| add SCCL during stage 1 (ablation) | SECap Q-Former · EMOSpeech test | SIM1: 3.14% | Helped |
Surprising findings
- Freezing the Q-Former in the second training stage drops SIM1 from 71.95 to 57.92, a 19.5% reduction.
- Adding raw or T-Embedding transcription alongside speech features lowers objective scores compared with speech features alone.
Limitations
- Evaluated only on an internal Chinese dataset (EMOSpeech) with 7 speakers.
- Subjective evaluation involves only 15 evaluators and 50 sentences.
- No existing evaluation method for speech emotion captions; metrics adapted from audio captioning may not capture emotion fidelity.
Experimental setup
Datasets: EMOSpeech WenetSpeech
Benchmarks: EMOSpeech
Metrics: SIM1 SIM2 BLEU1 BLEU4 METEOR ROUGE-L CIDEr SPICE MOS
Baselines: HTSAT-BART Human Label SER Model Label
Builds on: BLIP-2 CLAP vCLUB HTSAT-BART
Closest papers by meaning
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.85)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.84)
- EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning (ICLR 2026 · similarity 0.84)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.83)
- OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.83)
- SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.