Research map / Large language models / Language, safety and interpretability
InstructSpeech: Following Speech Editing Instructions via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a triplet (instruction, input speech, output speech) dataset and trains InstructSpeech, a multi-task speech-token language model with task embeddings, multi-step reasoning, and a hierarchical adapter for instruction-guided speech editing.
Problem
Instruction-guided speech editing of semantic and acoustic attributes is hard due to scarce triplet training data and the difficulty of accurately executing diverse natural-language editing instructions.
Approach
Casts speech editing as sequence-to-sequence generation over discrete speech tokens conditioned on frozen Flan-T5-XL instruction embeddings, with a predicted task tag, multi-task training on speech editing and processing tasks, chain-of-thought-style decomposition for free-form edits, and a hierarchical adapter for few-shot adaptation to new tasks.
Key results
- Region-based content editing reaches WER 5.6 averaged over add/remove/replace; Large model achieves emotion accuracy 57.3 and style accuracy 92.3 in acoustic editing.
- The hierarchical adapter updates about 1% of parameters (3.03M) and reaches WER 0.38 and SIM 0.85 in emotion few-shot adaptation, versus WER 0.32 for full finetuning and 0.41 for LoRA.
- Human evaluation reports MOS 4.01 and SMOS 3.95 for the medium model, above YourTTS (3.91 MOS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multitask learning across speech editing and processing tasks (ablation) | InstructSpeech · Region-based content editing | WER: 7.4 -> 5.9 | Helped |
| add multitask learning across speech editing and processing tasks (ablation) | InstructSpeech · Emotion editing | accuracy: 51.3 -> 55.2 | Helped |
| add multitask learning across speech editing and processing tasks (ablation) | InstructSpeech · Emotion editing | FFE: 0.43 -> 0.41 | Helped |
| add task embedding conditioning (ablation) | InstructSpeech · Task category prediction | accuracy: 73.2% -> 100% | Helped |
| replace ground-truth task embeddings with predicted task embeddings (ablation) | InstructSpeech · Task category prediction | accuracy: 100% -> 100% | No clear effect |
| use hierarchical adapter instead of full fine-tuning (ablation) | InstructSpeech · Emotion task fine-tuning | WER: 0.32 -> 0.38 | Hurt |
| use hierarchical adapter instead of full fine-tuning (ablation) | InstructSpeech · Emotion task fine-tuning | SIM: 0.86 -> 0.85 | Hurt |
| use hierarchical adapter instead of LoRA (ablation) | InstructSpeech · Emotion task fine-tuning | WER: 0.41 -> 0.38 | Helped |
| use hierarchical adapter instead of LoRA (ablation) | InstructSpeech · Emotion task fine-tuning | SIM: 0.84 -> 0.85 | Helped |
| scale model from 520M to 1.2B parameters | InstructSpeech · Acoustic editing | emotion accuracy: 55.2 -> 57.3 | Helped |
| scale model from 520M to 1.2B parameters | InstructSpeech · LibriSpeech-test, free-form content editing | WER (Add): 13.1 -> 12.6 | Helped |
| scale model from 520M to 1.2B parameters | InstructSpeech · LibriSpeech-test, free-form content editing | WER (Remove): 12.0 -> 11.9 | Helped |
Surprising findings
- Predicted task embeddings from the fine-tuned Flan-T5-XL match ground-truth embeddings in task accuracy (100%), while removing task conditioning drops accuracy to 73.2%.
Limitations
- Speed alteration only supports speech in the LJSpeech speaker identity.
- Free-form editing accuracy is limited (overall WER 12.6-15.0 for add/remove/replace in multi-step reasoning).
Experimental setup
Datasets: LibriLight LibriSpeech LibriTTS VCTK ESD LibriTTS-style
Benchmarks: LibriSpeech-test ESD InstructSpeech instruction-editing benchmark
Metrics: WER PER SIM FFE accuracy CISP text-speech similarity CISP speech similarity MOS SMOS retrieval R@1/5/10
Baselines: EditSpeech A3T YourTTS GT (voc.)
Builds on: VALL-E-style decoder-only speech LM (Megabyte-style multiscale transformer) HuBERT SoundStream Flan-T5 BigVGAN CLAP
Closest papers by meaning
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions (NeurIPS 2025 · similarity 0.90)
- Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR 2024 · similarity 0.88)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.87)
- Enhancing Speech Large Language Models through Reinforced Behavior Alignment (NeurIPS 2026 · similarity 0.86)
- #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models (ICLR 2024 · similarity 0.86)
- Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.