Research map / Large language models / Instruction and fine-tuning

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 19 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Contrastive Preference Optimization (CPO), a reference-free-data preference method that trains ALMA-13B with triplet-based preference data from FLORES-200 to produce ALMA-R, matching or exceeding GPT-4 and WMT winners on WMT'21/'22/'23 translation.

Problem

Moderate-sized (7B/13B) LLM-based machine translation lags behind GPT-4 and WMT competition winners, and supervised fine-tuning on gold references caps performance at reference quality, which is itself often flawed.

Approach

CPO combines a DPO-style preference loss with a uniform reference model (removing the reference-model forward pass) plus an NLL term on preferred translations, using preference triplets (reference, GPT-4, ALMA outputs) ranked by reference-free metrics so the model learns to avoid good-but-imperfect translations.

Key results

• ALMA-13B-R reaches 85.74 KIWI-XXL and 94.05 XCOMET average on en→xx WMT'21/'22, above GPT-4 (83.83 / 93.23) and WMT winners (84.81 / 93.78).
• On xx→en, ALMA-13B-R averages 82.43 KIWI-XXL and 89.11 XCOMET, versus 82.60 / 89.41 for GPT-4 on KIWI-XXL/XCOMET (near parity, slightly below GPT-4 on the averages shown).
• DPO and SFT on the same preferred data slightly degrade or barely change ALMA-13B-LoRA performance, while CPO gives consistent gains; in human evaluation on zh→en, ALMA-13B-R scored 5.16 vs 4.86 for ALMA-13B-LoRA.

Measured effects reported in the paper

Surprising findings

• Gold FLORES-200 references are often scored below model outputs by reference-free metrics, with ALMA-13B-LoRA and GPT-4 beating the reference in a majority of xx→en cases (e.g., 73% for ALMA on KIWI-XXL).
• Plain DPO and SFT on metric-preferred data can lower scores on the very metric used to select the data, while CPO improves them.

Limitations

• Preference labels come from reference-free neural metrics and GPT-4 outputs, so gains could partly reflect metric bias (mitigated but not eliminated).
• Human-labeled preference data covers only two directions and was found to have minimal effect.
• Evaluation is limited to English-centric pairs for 5 languages and mostly reference-free metrics; BLEU and COMET-22 show inconsistencies.

Experimental setup

Datasets: FLORES-200 WMT'21 test WMT'22 test WMT'23 test

Benchmarks: WMT'21 WMT'22 WMT'23

Metrics: KIWI-XXL XCOMET KIWI-22 COMET-22 BLEU sacreBLEU BLEURT human score (0-6)

Baselines: ALMA-13B-LoRA GPT-4 WMT competition winners TowerInstruct SFT on preferred data DPO NLLB-3.3B MADLAD-10B Google Translate

Builds on: DPO ALMA LoRA SFT behavior cloning regularization

Closest papers by meaning

• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.91)
• X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale (ICLR 2025 · similarity 0.89)
• Calibrating Translation Decoding with Quality Estimation on LLMs (NeurIPS 2025 · similarity 0.88)
• Teaching Large Language Models to Translate with Comparison (AAAI 2024 · similarity 0.88)
• SimulPL: Aligning Human Preferences in Simultaneous Machine Translation (ICLR 2025 · similarity 0.88)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)