Research map / Architectures and efficiency / Attention and transformers

Zipformer: A faster and better encoder for automatic speech recognition

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Efficient language transformers · 28 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Zipformer, a U-Net-like, downsampled Transformer encoder for ASR with BiasNorm, SwooshR/SwooshL activations, and the ScaledAdam optimizer, reporting SOTA-level WER on LibriSpeech, Aishell-1, and WenetSpeech.

Problem

Conformer encoders operate at a constant frame rate and are costly in compute and memory; Adam-type optimizers do not account for parameter scale, and LayerNorm and Swish have training issues in ASR encoders.

Approach

A U-Net-like encoder that downsamples to lower frame rates across stacks, a block with reused attention weights (MHAW shared with NLA and SA modules), BiasNorm in place of LayerNorm, Swoosh activations, and ScaledAdam, which scales updates by parameter RMS and learns the parameter scale explicitly.</modelcore_idea>

Key results

• Zipformer-L (pruned transducer) reaches 2.06%/4.63% WER on LibriSpeech test-clean/test-other, and 2.00%/4.38% when trained 170 epochs on 8 A100s.
• Zipformer-L achieves 4.28% CER on Aishell-1 test, and Zipformer-L obtains 7.24%/12.06% on WenetSpeech Test_Net/Test_Meeting.
• In ablation on Zipformer-M, ScaledAdam beats Adam by 0.17% on test-clean and 0.72% on test-other.

Measured effects reported in the paper

Surprising findings

• Removing the Whitener slightly improves test-clean WER (2.21 to 2.25 is worse, but test-other reported differently), so activation constraints show mixed effects
• Zipformer-M with a Conformer-style block doubled in count reaches only a 0.16% test-other gain, implying block-structure gains are modest

Limitations

• Pruned transducer loss only; experiments mainly on three datasets
• Some activation constraints (Balancer/Whitener) are placed empirically without clear rules
• Hyperparameters like the Eden schedule and Bypass limits are heuristic

Experimental setup

Datasets: LibriSpeech Aishell-1 WenetSpeech

Benchmarks: LibriSpeech test-clean LibriSpeech test-other Aishell-1 WenetSpeech Dev WenetSpeech Test_Net WenetSpeech Test_Meeting

Metrics: WER CER parameters GFLOPs inference time peak memory

Baselines: Conformer Squeezeformer Branchformer E-Branchformer Conformer-MoE Adam

Builds on: Conformer Adam Squeezeformer pruned transducer Swish

Closest papers by meaning

• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition (NeurIPS 2022 · similarity 0.93)
• Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.89)
• Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
• Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers (NeurIPS 2024 · similarity 0.87)
• Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation (ICLR 2025 · similarity 0.85)
• Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping (NeurIPS 2025 · similarity 0.85)