AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

FiRE: Fine-grained Ranking Evaluation for Machine Translation

Wenyang Gao, Yinghao Yang, Xi Jin, Jing Li, Yue Zhang

ICML 2026 · Large language models › Language, safety and interpretability · approach: Controlled text generation

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use criterion-conditioned FiRE aggregation instead of a direct overall ranking prompt (ablation)DeepSeek-R1 · Overall pairwise ranking EN→ZHagreement with human (%): 64.3 -> 65.3Improved
use criterion-conditioned FiRE aggregation instead of a direct overall ranking prompt (ablation)DeepSeek-R1 · Overall pairwise ranking RU→ZHagreement with human (%): 66.7 -> 70.1Improved
use LLM pairwise FiRE instead of error-based evaluators on fluency and styleMT evaluation · Consistency of style EN→ZHagreement with human (%): 3.0 -> 61.4 (GEMBA-MQM vs FiRE)Improved
use LLM-based FiRE instead of trained regression and ranking metricsMT evaluation · Overall pairwise ranking EN→ZHagreement with human (%): 61.6 -> 65.3 (MetricX-24-XXL vs FiRE)Improved
evaluate harder examples (two of three annotators agree) rather than easy ones (ablation)LLM evaluators · Faithfulness EN→ZHagreement with human (%): 57.6 -> 43.4Hurt
swap candidate order for dedicated ranker (ablation)MT-Ranker-XXL · Overall RU→ZHagreement with human (%): 61.6 -> 41.9Hurt
swap candidate order with FiRE aggregation (ablation)DeepSeek-R1-FiRE · Overall EN→ZHagreement with human (%): 65.3 -> 65.9No clear effect
reconcile judgments at criterion level before aggregation versus collapsing each order to overall decision first (ablation)FiRE · Position bias mitigationagreement with humanImproved
add locale convention as a fourth criterion (FiREplus)FiRE · 250 new EN→ZH pairsoverall agreementImproved
add FiREplus criterion on full benchmarkFiRE · EN→ZH and RU→ZH full benchmarkoverall agreementNo clear effect
evaluate with same-lineage LLM judgeGPT-4o and Qwen2.5-72B evaluators · EN→ZH and RU→ZHself-preferenceHurt
use reasoning models (DeepSeek-R1, QwQ-32B) as evaluatorsFiRE · Position consistency EN→ZHposition consistencyImproved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy