Research map / Architectures and efficiency / Attention and transformers

AEQA-NAT : Adaptive End-to-end Quantization Alignment Training Framework for Non-autoregressive Machine Translation

ICML 2025 · Architectures and efficiency › Attention and transformers · approach: Efficient language transformers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AEQA-NAT, a non-autoregressive translation framework that uses a pre-aligned semantic quantization space (VQ-VAE-style, initialized from mBART) to remove the target-information dependency of MLM/glancing training and close the training-inference gap.

Problem

Non-autoregressive translation models trained with masked language modeling / glancing sample target tokens as decoder input, creating a training-inference gap because target tokens are unavailable at inference, which limits NAT quality.

Approach

Replace target-token conditioning with a shared semantic quantization space (vector-quantized codebook, EMA-updated and pre-aligned using mBART) that the encoder and decoder outputs are constrained to, plus aligned reordering of decoder inputs, so training and inference use the same source-only inputs.

Key results

• AEQA-NAT reaches 26.87 BLEU on WMT14 EN-DE raw data at 17.0x speedup (batch size 1), and AEQA-DAT reaches 27.03 BLEU at 9.1x speedup, versus DAT's 26.57 raw BLEU.
• The gap between raw-data and distilled-data training for AEQA-NAT is 0.29 BLEU on WMT14 EN-DE, versus 0.81 for DAT and 7.24 for Vanilla NAT.
• Adding the semantic quantization space improves BLEU by more than 9 points over the baseline ablation (Table 6, line 1 vs. line 2).

Measured effects reported in the paper

Surprising findings

• Simply annealing the GLAT sampling rate to zero during training degrades performance, as reported from prior work and motivating the approach.
• Training on raw data without distillation gives AEQA-NAT nearly the same BLEU as with distillation (0.29 gap), in contrast to Vanilla NAT's 7.24 gap.

Limitations

• Performance remains below the autoregressive Transformer on several directions (gap of up to about 1 BLEU on WMT14 EN-DE), and NPD-based variants lower speed.
• Relies on an mBART model to pre-align the semantic quantization space, adding a dependency on a large pretrained model.
• The sampling ratio lambda is dataset-dependent and must be tuned.

Experimental setup

Datasets: WMT14 EN-DE WMT16 EN-RO WMT17 ZH-EN IWSLT16 DE-EN multi-reference WMT14 EN-DE test set (Ott et al. 2018)

Benchmarks: WMT14 EN-DE WMT14 DE-EN WMT16 EN-RO WMT16 RO-EN WMT17 ZH-EN IWSLT16 DE-EN

Metrics: BLEU SacreBLEU chrF COMET BLEURT speedup n-gram repetition

Baselines: Transformer (AT) Vanilla NAT CMLM GLAT DAT CTC AXE OaXE DSLP Latent-GLAT

Builds on: Glancing Transformer Conditional Masked Language Model Directed Acyclic Transformer VQ-VAE mBART

Closest papers by meaning

• Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation (NeurIPS 2022 · similarity 0.90)
• Improving Non-Autoregressive Translation Models Without Distillation (ICLR 2022 · similarity 0.90)
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input (AAAI 2019 · similarity 0.90)
• DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (NeurIPS 2024 · similarity 0.87)
• Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.87)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.87)