Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EBBS, an ensemble decoding method with bi-level beam search that combines direct and pivot zero-shot translations via soft voting, and distills the ensemble back into the multilingual model.
Problem
Zero-shot machine translation in multilingual models is noisy when translating directly, and pivoting through English accumulates errors; existing ensembles (word-level averaging/voting, MBR) do not combine them effectively.
Approach
Bi-level beam search: each ensemble component runs its own beam search at the lower level, while an upper-level shared beam is formed by summing each candidate's probabilities across components' beams (soft voting), avoiding probability averaging that over-smooths modes. The ensemble output is then distilled into the multilingual model for efficient direct translation.
Key results
- EBBS averages 18.24 BLEU vs 17.29 for direct translation on IWSLT and 28.36 vs 27.74 on Europarl.
- EBBS beats direct translation in 56 of 62 zero-shot directions across both datasets (two-sided binomial p = 3e-11).
- EBBS-distilled direct model reaches 18.92 BLEU on IWSLT and 28.54 on Europarl, outperforming EBBS itself while having no inference overhead over direct translation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EBBS instead of direct translation | Multilingual translation model · IWSLT zero-shot directions (average) | BLEU: 17.29 -> 18.24 | Helped |
| Use EBBS instead of direct translation | Multilingual translation model · Europarl all zero-shot directions | BLEU: 27.85 -> 28.44 | Helped |
| Distill EBBS outputs into the direct-translation model | Direct-translation model · Italian-to-Dutch (IWSLT) | BLEU: 17.46 -> 20.13 | Helped |
| Distill EBBS outputs instead of using no distillation | Direct-translation model · Italian-to-Dutch (Europarl) | BLEU: 25.33 -> 25.92 | Helped |
| Distill the model's own direct-translation predictions | Direct-translation model · Italian-to-Dutch (IWSLT) | BLEU: 17.46 -> 18.10 | Helped |
| Train on the union of direct and pivot teachers' outputs | Direct-translation model · Italian-to-Dutch (IWSLT) | BLEU: 17.46 -> 17.80 | Helped |
| Train on the union of direct and pivot teachers' outputs | Direct-translation model · Italian-to-Dutch (Europarl) | BLEU: 25.33 -> 25.53 | Helped |
| Use beam size 1 at both levels instead of beam size 5 (ablation) | EBBS · Italian-to-Dutch (Europarl) | BLEU: 26.10 -> 24.98 | Hurt |
| Use beam size 3 at both levels instead of beam size 5 (ablation) | EBBS · Italian-to-Dutch (Europarl) | BLEU: 26.10 -> 25.99 | Hurt |
| Use beam size 9 at both levels instead of beam size 5 (ablation) | EBBS · Italian-to-Dutch (Europarl) | BLEU: 26.10 -> 26.12 | No clear effect |
| Replace top-Z sum voting with total-sum voting (ablation) | EBBS · Italian-to-Dutch (Europarl) | BLEU: 26.10 -> 25.27 | Hurt |
| Replace top-Z sum voting with max voting (ablation) | EBBS · Italian-to-Dutch (Europarl) | BLEU: 26.10 -> 25.81 | Hurt |
Surprising findings
- Pivoting through English does not outperform direct translation in this setting.
- Word-level averaging ensemble performs roughly the same as direct translation, unlike prior reports for same-path random-seed ensembles.
Limitations
- Inference cost scales linearly with the number of ensemble components.
- Gains on IWSLT for MBR and word-level ensembles are negligible, and EBBS is worse than direct on one Europarl direction for distillation (Danish-to-Dutch).
- Distillation experiments were run on a subset of directions (Danish-to-others) due to compute cost.
Experimental setup
Datasets: IWSLT 2017 Europarl v7
Benchmarks: IWSLT 2017 Europarl v7
Metrics: BLEU chrF2++ TER COMET
Baselines: Direct translation Pivoting (en) Word-level averaging ensemble Word-level voting ensemble Sequence-level voting ensemble (MBR) Direct distillation Union distillation
Builds on: beam search English-centric multilingual NMT sequence-level knowledge distillation minimum Bayes risk decoding
Closest papers by meaning
- Transductive Ensemble Learning for Neural Machine Translation (AAAI 2020 · similarity 0.86)
- QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation (NeurIPS 2024 · similarity 0.86)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.85)
- Calibrating Translation Decoding with Quality Estimation on LLMs (NeurIPS 2025 · similarity 0.85)
- X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale (ICLR 2025 · similarity 0.84)
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.