Research map / Large language models / Language, safety and interpretability
Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC
Linjuan Wu, Ruiqi Zhang, Xinze Lyu, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Yixin Cao, Yongliang Shen, Weiming Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Qwen3-8B on JUDGER-refined UGC translation data | Qwen3-8B · CULTURE-MT | ineffective share: 49.60% -> 14.47%; score-3 share 4.09% -> 28.24% | Improved |
| fine-tune Qwen3-32B on JUDGER-refined UGC translation data | Qwen3-32B · CULTURE-MT | ineffective share: 30.74% -> 13.97% | Improved |
| add JUDGER-guided rewrite loop to training data construction (ablation) | Ours-32B · CULTURE-MT | ineffective share: 16.87% -> 13.97% | Improved |
| add JUDGER-guided rewrite loop to training data construction (ablation) | Ours-8B · CULTURE-MT | ineffective share: 16.17% -> 14.47% | Improved |
| scale base LLM size (Qwen3 0.6B to 32B) | Qwen3 family · CULTURE-MT | ineffective share: 95.41% (0.6B) -> 30.74% (32B) | Improved |
| use BLEU/ChrF/COMET instead of cultural effectiveness to compare scales | evaluation metrics · CULTURE-MT | metric discriminability | No clear effect |
| fine-tune JUDGER (Qwen3-32B SFT) vs prompted Qwen3-32B | Qwen3-32B judge · Judger validation set | accuracy: 78.05% -> 86.03% | Improved |
| use translation-specialized Seed-X-PPO vs general Qwen3-32B-sized models | Seed-X-PPO · CULTURE-MT | average cultural score: 1.71 avg vs Qwen3-32B 1.73 | No clear effect |
| fine-tune Qwen3-8B on UGC data, evaluate on general FLORES | Qwen3-8B · FLORES | BLEU / COMET: BLEU 58.06 -> 15.92; COMET 87.38 -> 84.47 | Hurt |
| increase symbol density and expressiveness of notes (General to Symbol/Hybrid) | most models · CULTURE-MT note types | ineffective share | Hurt |
Closest papers by meaning
- CulturePark: Boosting Cross-cultural Understanding in Large Language Models (NeurIPS 2024 · similarity 0.87)
- CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.87)
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.86)
- BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.85)
- Beyond Aesthetics: Cultural Competence in Text-to-Image Models (NeurIPS 2024 · similarity 0.84)
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
