Research map / Optimization / Optimizers
Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes objective soup recipes (VS-MSP, VC-MSP, VM-MSP) that apply multi-objective optimization at single, bilevel and multilevel granularity to jointly train multilingual ASR and speech translation, with layer-selection to cut overhead.
Problem
Training one multilingual multi-task speech processing model for ASR and speech-to-text translation suffers from conflicting gradients among objectives, and MOO becomes less effective as the number of tasks grows.
Approach
Place self-supervised CPC loss as a lower-level penalty/constraint and separate highly conflicting ASR (CTC) and translation (CE) objectives into hierarchical optimization levels, using MoDo-style dynamic conflict-avoiding weights computed only on conflicting layers.
Key results
- VC-MSP outperforms the two-stage baseline by 9.8% WER and 8.6% BLEU (as stated in the paper's findings).
- VM-MSP (UAS) reaches average WER 18.4% on the 150M Conformer on CoVoST 2 vs 20.5% for two-stage training.
- Layer-selection efficient variant reduces GPU memory from 14.7 GB to 12.8 GB and epoch time from 4.1 h to 3.7 h while keeping accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace two-stage training with VC-MSP | Conformer MSP · CoVoST-2, 150M Conformer, 5-language average | WER: 9.8% | Helped |
| Use multilevel VM-MSP rather than VC-MSP | Conformer MSP · CoVoST-2, 150M Conformer, 5-language average | WER: 4.2% | Helped |
| Use multilevel VM-MSP rather than VC-MSP | Conformer MSP · CoVoST-2, 150M Conformer, 4-language average | BLEU: 2.8% | Helped |
| Use UAS rather than USA task order (ablation) | VM-MSP · CoVoST-2 Conformer and Whisper-medium speech recognition | WER | Helped |
| Use USA rather than UAS task order (ablation) | VM-MSP · CoVoST-2 Conformer and Whisper-medium translation | BLEU | Helped |
| Compute dynamic update direction only on conflicting layers (ablation) | VM-MSP · CoVoST-2 Conformer and Whisper-medium | WER and BLEU: 14.7 GB to 12.8 GB; 4.1 h to 3.7 h per epoch | No clear effect |
| Increase penalty parameter rate from 0.002 to 0.02 per epoch (ablation) | VM-MSP · CoVoST-2 Conformer 83M | WER and BLEU: 0.002; 0.02 | Mixed |
Surprising findings
- Self-supervised gradients show higher alignment with the other objectives, so they serve well as a lower-level constraint.
- Optimization order matters: UAS (self-supervised, ASR, translation) gives better recognition than USA.
Limitations
- Evaluated only on architectures with a shared encoder and task-specific heads; generalization to a shared unified decoder is untested.
- Penalty parameter selection involves trade-offs between upper- and lower-level performance.
- Language and task hierarchies tested on a limited set of languages (5 for ASR, 4 for translation).
Experimental setup
Datasets: CoVoST 2 LibriSpeech AISHELL-1
Benchmarks: CoVoST 2 LibriSpeech AISHELL-1
Metrics: WER BLEU
Baselines: two-stage training two-stage static weighting joint training VS-MSP VC-MSP Whisper-medium
Builds on: MoDo MGDA CPC Conformer Whisper
Closest papers by meaning
- OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models (ICML 2025 · similarity 0.86)
- LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration (AAAI 2025 · similarity 0.86)
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.86)
- SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.86)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.85)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.