Research map / Large language models / Language, safety and interpretability
FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends Qwen2-Audio to long-speech understanding by adding a CTC-guided iterative fusion speech extractor and dynamic-compression LoRA training on short-speech data, plus a new LongSpeech-Eval benchmark.
Problem
Large speech-language models are largely limited to short clips (under ~30 s), and long-speech training data is scarce while long speech sequences are computationally expensive for LLMs.
Approach
An extractor uses a CTC decoder's non-blank probability as content density and adjacent-frame cosine similarity to iteratively merge speech frames down to a target length; LLM is then trained with dynamic compression ratios sampled from a set of target lengths, using only short-speech data.
Key results
- On LongSpeech-Eval, FastLongSpeech scores 3.55 versus 3.44 for NTK-RoPE, with 26.44 TFLOPs and 1.47 s runtime versus 61.21 TFLOPs and 4.80 s for NTK-RoPE.
- On the SPIRAL-H benchmark, FastLongSpeech (L=750) reaches 76.81% accuracy at a 65.88% prune rate, versus 63.77% for SpeechPrune at 60.00%.
- On MELD at a 50% inference-cost reduction, FastLongSpeech achieves 52.95% accuracy versus 52.14% for FastAdaSP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FastLongSpeech rather than NTK-RoPE | Qwen2-Audio-7B-Instruct · LongSpeech-Eval | score: 3.44 -> 3.55 | Helped |
| Use FastLongSpeech rather than NTK-RoPE | Qwen2-Audio-7B-Instruct · LongSpeech-Eval | TFLOPs: 61.21 -> 26.44 | Helped |
| Use FastLongSpeech rather than NTK-RoPE | Qwen2-Audio-7B-Instruct · LongSpeech-Eval | inference time: 4.80 -> 1.47 | Helped |
| Use FastLongSpeech rather than vanilla Qwen2-Audio | Qwen2-Audio-7B-Instruct · LibriTTS test subset of OpenASQA | TFLOPs: 9.79 -> 5.64 | Helped |
| Replace dynamic compression training with standard fine-tuning (ablation) | FastLongSpeech · LongSpeech-Eval | score: 3.55 -> 3.33 | Hurt |
| Add multiple iterations to iterative fusion (ablation) | FastLongSpeech · LongSpeech-Eval | score: 3.41 -> 3.55 | Helped |
| Add content-density-weighted merging instead of average pooling (ablation) | FastLongSpeech · LongSpeech-Eval | score: 3.28 -> 3.55 | Helped |
| Increase context length from L=200 to L=750 (ablation) | FastLongSpeech · LongSpeech-Eval | score: 2.47 -> 3.55 | Helped |
| Extend context length from L=750 to L=1200 (ablation) | FastLongSpeech · LongSpeech-Eval | score: 3.55 -> 3.66 | Helped |
| Extend context length from L=1200 to L=4000 (ablation) | FastLongSpeech · LongSpeech-Eval | score: 3.66 -> 3.59 | Hurt |
| Apply iterative fusion at L=400 to vanilla Qwen2-Audio | Qwen2-Audio · speech_dialogue_QA_fisher | score: 3.95 -> 4.13 | Helped |
| Apply iterative fusion at L=200 to vanilla Qwen2-Audio | Qwen2-Audio · speech_dialogue_QA_fisher | TFLOPs: 11.76 -> 5.46 | Helped |
Surprising findings
- Applying the iterative fusion directly to vanilla Qwen2-Audio without any extra training at L=400 slightly exceeds baseline score on the dialogue task (4.13 vs 3.95).
- The Cascaded pipeline achieves the highest long-speech score (3.75) but is far slower (17.23+1.38 s) than FastLongSpeech.
Limitations
- Only short-speech training data is used; performance on very long inputs beyond training range (L=4000) slightly declines.
- Aggressive compression (L=100) degrades ASR substantially (WER 27.12 on LibriSpeech test-clean), so the best compression ratio is task-dependent.
- LongSpeech-Eval is small (164 samples) and synthetic, with LLM-judge scoring.
Experimental setup
Datasets: LibriSpeech MLS OpenASQA LibriSQA Common Voice LongBench (MultiFieldQA-En, NarrativeQA) AIR-Bench speech_QA_iemocap AIR-Bench speech_dialogue_QA_fisher LibriTTS test subset (OpenASQA) MELD GigaSpeech SPIRAL-H
Benchmarks: LongSpeech-Eval AIR-Bench speech_QA_iemocap LibriSQA test OpenASQA LibriTTS test subset AIR-Bench speech_dialogue_QA_fisher MELD LibriSpeech test-clean/test-other GigaSpeech test SPIRAL-H
Metrics: LLM-judge score (1-5) WER accuracy TFLOPs runtime (s)
Baselines: Random AvgPool MostSim NTK-RoPE Cascaded (Whisper-Large-V3 + Qwen-7B-Chat) FastAdaSP SpeechPrune Baseline Qwen2-Audio
Builds on: Qwen2-Audio Connectionist Temporal Classification NTK-aware scaled RoPE LongBench
Closest papers by meaning
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.91)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.89)
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.89)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.88)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.88)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.