Research map / Architectures and efficiency / Attention and transformers
Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCformer, a Transformer design that pairs a high-order (RK2/RK4) predictor with EMA-parameterized coefficients and a multistep or backward-Euler corrector, improving machine translation, summarization, LM and GLUE results over ODE Transformer baselines.
Problem
Residual Transformer blocks act as first-order Euler discretizations, leaving truncation error; prior high-order ODE Transformers rely on gated coefficient learning that does not scale to higher orders (e.g., RK4) and stabilizes poorly.
Approach
Combine a high-order explicit predictor with a multistep/implicit corrector (predictor-corrector paradigm), and replace constant or gated coefficients with an exponential-moving-average parameterization (gamma*(1-gamma)^k) with a learnable gamma, plus RK-Norm for stability.
Key results
- PCformer reaches 30.95 BLEU on WMT'14 En-De and 44.27 BLEU on En-Fr with the 12-6 big configuration.
- A 12-layer PCformer (2-order) reaches 32.6 average SacreBLEU on OPUS-100 with fewer parameters than the 3.8B DeepNet (31.1 to 32.1 range reported for DeepNet).
- On LM Evaluation Harness, PCformer at 1.3B improves average accuracy over Transformer++ by 5.7 points (per paper text); GLUE average rises to 86.1 vs BERT 84.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace gated coefficient learning with EMA coefficient learning | RK2-block · WMT'14 En-De | BLEU: 28.89 -> 29.11 | Helped |
| add the predictor-corrector framework | high-order Transformer · WMT'14 En-De | BLEU: 30.77 -> 30.95 | Helped |
| add the predictor-corrector framework | RK4-block · CNN/DailyMail | ROUGE-1: 41.83 -> 42.10 | Helped |
| set EMA coefficient γ to 0.50 rather than 0.25, 0.75, or 0.99 (ablation) | RK4-block (EMA) · WMT'14 En-De | BLEU: 28.90; 29.43; 28.99; 29.20 | Helped |
| remove RK-Norm (ablation) | PCformer · WMT'14 En-De | BLEU | Hurt |
| use layer-wise γ instead of scalar γ (ablation) | PCformer · WMT'14 En-De | BLEU: 30.95 -> 30.63 | No clear effect |
| replace scalar γ with vector γ (ablation) | PCformer · WMT'14 En-De | BLEU: 30.95 -> 30.88 | No clear effect |
| use a multistep predictor with an RK2-block EMA corrector (ablation) | predictor-corrector framework · WMT'14 En-De | BLEU: 30.70 -> 29.78 | Hurt |
| use a multistep predictor with a Backward Euler corrector (ablation) | predictor-corrector framework · WMT'14 En-De | BLEU: 30.70 -> 30.30 | Hurt |
| use a multistep predictor and multistep corrector instead of an RK2-block EMA predictor (ablation) | predictor-corrector framework · WMT'14 En-De | BLEU: 30.70 -> 30.30 | Hurt |
| use Backward Euler rather than a multistep corrector (ablation) | RK2-block with EMA predictor · WMT'14 En-De | BLEU: 30.70 -> 30.95 | Helped |
Surprising findings
- Learned unconstrained coefficients in RK2/RK4 blocks drift to values resembling Adams-Bashforth coefficients, including negative weights, while EMA constraint yields better results.
- Removing RK-Norm causes training to fail entirely.
Limitations
- High-order predictors add substantial inference overhead in decoder-only and encoder-only settings.
- Backward Euler corrector preferred on small datasets since more complex correctors overfit.
- Several main results averaged over three seeds only; code release is promised but not yet available at submission.
Experimental setup
Datasets: WMT'14 En-De WMT'14 En-Fr WMT'16 En-Ro OPUS-100 CNN/DailyMail Penn Treebank Wikitext-103 SlimPajama GLUE UEA time series archive
Benchmarks: WMT'14 En-De WMT'14 En-Fr WMT'16 En-Ro OPUS-100 CNN/DailyMail Wikitext-103 Penn Treebank GLUE LM Evaluation Harness UEA time series
Metrics: BLEU SacreBLEU COMET ROUGE perplexity accuracy Matthews correlation Pearson correlation
Baselines: Transformer-Big ODE Transformer (RK2 gated) RK4-block DeepNet BranchNorm DeLight Transformer++ Flowformer BERT Adaptive Input Transformer Shortformer
Builds on: ODE Transformer Dynamic Linear Combination of Layers (DLCL) Adams-Bashforth-Moulton methods DeepNet Runge-Kutta methods
Closest papers by meaning
- Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning (ICLR 2025 · similarity 0.88)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.85)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.85)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.84)
- How do Transformers Perform In-Context Autoregressive Learning ? (ICML 2024 · similarity 0.84)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.