AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap

Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, Björn Wolfgang Schuller

TPAMI 2023 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 472 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune the transformer layers instead of freezing them (ablation)wav2vec2 / HuBERT SER models · MSP-Podcast, IEMOCAP, MOSICCCImproved
use pre-trained weights instead of random initialisation (ablation)wav2vec2-large · valence / sentiment, in-domain and cross-corpusCCCImproved
use pre-trained weights instead of random initialisation (ablation)wav2vec2-large · arousal and dominanceCCCImproved
fuse explicit BERT text embeddings with fine-tuned audio embeddingsfine-tuned wav2vec2 / HuBERT · MSP-Podcast, IEMOCAP, MOSICCCMixed
pre-train on more, multilingual data (voxpopuli 100k h to xls-r 463k h)wav2vec2 large · MSP-Podcast, IEMOCAP, MOSICCC: 100k and 463k vs 63k hoursHurt
use large architecture instead of basewav2vec2 · MSP-Podcast, IEMOCAP, MOSICCCNo clear effect
use HuBERT instead of wav2vec2 (base)base architecture · MSP-Podcast, IEMOCAP, MOSI valencevalence CCCImproved
remove the top 12 of 24 transformer layers before fine-tuning (ablation)wav2vec2-large-robust · MSP-Podcast, IEMOCAP, MOSICCC: 24 -> 12 layersNo clear effect
keep 10 or fewer transformer layers (ablation)wav2vec2-large-robust · IEMOCAP and MOSI valencevalence / sentiment CCC: 10 or fewer layersHurt
reduce fine-tuning data to 25% of MSP-Podcast training set (ablation)pruned wav2vec2-large-robust · arousal and dominance on MSP-PodcastCCC: loss of less than .02 at 25%No clear effect
use the full training set rather than a reduced fraction (ablation)pruned wav2vec2-large-robust · valence on MSP-Podcast and MOSIvalence CCC: boost of almost .1 on MOSI for the remaining 25%Improved
add background noise at 0 dB instead of 20 dB SNRwav2vec2-large-robust · MSP-Podcast valencechange in CCC: -.221 vs -.003 (natural sounds)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy