Research map / Architectures and efficiency / Attention and transformers
Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Temporal U-Net structure (ablation) | Conformer-CTC-M · LibriSpeech test-other | WER (%): 7.90 -> 7.28 | Helped |
| replace FMCF/Macaron blocks with Transformer-style MF/CF blocks (ablation) | Temporal U-Net model · LibriSpeech test-other | WER (%): 7.28 -> 7.12 | Helped |
| replace GLU with Swish in the convolution module (ablation) | Transformer-style block model · LibriSpeech test-other | WER (%): 7.12 -> 7.09 | No clear effect |
| replace redundant pre-LayerNorm with learned scaling and scaled post-LayerNorm (ablation) | Unified-activation model · LibriSpeech test-other | WER (%): 7.09 -> 6.89 | Helped |
| replace the second subsampling convolution with depthwise separable convolution (ablation) | Simplified-LayerNorm model · LibriSpeech test-other | WER (%): 6.89 -> 6.89 | No clear effect |
| scale up model depth and width (ablation) | Squeezeformer-SM · LibriSpeech test-other | WER (%): 6.89 -> 6.50 | Helped |
| add the skip connection between downsampling and upsampling (ablation) | Squeezeformer-M · LibriSpeech dev-other | WER (%): 7.38 -> 6.51 | Helped |
| add the temporal upsampling layer (ablation) | Squeezeformer-M · LibriSpeech dev-other | WER (%) | Helped |
| add learned scaling instead of using PostLN only (ablation) | Squeezeformer-M · LibriSpeech dev-other | WER (%): 14.00 -> 6.51 | Helped |
| add learned scaling instead of using PreLN only (ablation) | Squeezeformer-M · LibriSpeech dev-other | WER (%): 8.27 -> 6.51 | Helped |
| replace GLU with Swish rather than removing the activation (ablation) | Squeezeformer-M convolution module · LibriSpeech dev-other | WER (%): 6.73 -> 6.51 | Helped |
| use Squeezeformer-L instead of Conformer-CTC-L | LibriSpeech ASR · LibriSpeech test-other | WER (%): 6.55 -> 5.97 | Helped |
Closest papers by meaning
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.93)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.88)
- Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers (NeurIPS 2024 · similarity 0.88)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
- On the Power of Convolution-Augmented Transformer (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.