Research map / Large language models / Agents, code and math
DiscoX: Benchmarking Discourse-Level Translation in Expert Domains
Xiying ZHAO, Zhoufutu Wen, Zhixuan Chen, Jingzhe Ding, Jianpeng Jiao, Shuai Li, Xi Li, Danni Liang, Shengda Long, Qianqian Liu, Xianbo Wu, Hongwan Gao and 10 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable thinking mode (ablation) | Qwen-3-235B · DiscoX | overall score: 59.66 -> 49.97 | Hurt |
| enable thinking mode (ablation) | Claude-4 · DiscoX | overall score: 54.04 -> 53.53 | No clear effect |
| remove error de-duplication from Metric-S (ablation) | Metric-S · human consistency, DiscoX | average consistency: 66.00% -> 70.30% | Improved |
| use a multi-agent workflow instead of a single LLM judge (ablation) | Metric-S · human consistency, DiscoX | average consistency: 24.90% -> 70.30% | Improved |
| add fluency and appropriateness dimensions to accuracy-only judging (ablation) | Metric-S · human consistency, DiscoX | average consistency: 62.50% -> 70.30% | Improved |
| use 60/20/20 dimension weights instead of equal weights (ablation) | Metric-S · human consistency, DiscoX | average consistency: 48.20% -> 70.30% | Improved |
| use Gemini-2.5-Pro as judge instead of DeepSeek-R1 or o3-high (ablation) | Metric-S · human consistency, DiscoX | average consistency: 57.8% / 58.2% -> 70.3% | Improved |
| o3 acting as judge shows self-preference (ablation) | Metric-S with o3-high judge · DiscoX samples | judge ranking: human 51.70 vs o3 judge 77.54 for o3-high | Hurt |
| compare translation direction en->zh vs zh->en | LLMs · DiscoX | overall score: DeepSeek-V3 gap 34.8 points; Doubao-1.6-T gap 7.2 | Hurt |
| repeat evaluation in three independent runs (ablation) | DiscoX · DiscoX | overall score: GPT-5: 76.66, 77.02, 77.03 | No clear effect |
Closest papers by meaning
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.86)
- A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR 2024 · similarity 0.85)
- X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale (ICLR 2025 · similarity 0.85)
- DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory (ICLR 2025 · similarity 0.85)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.85)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
