Research map / Image generation / Text-to-image diffusion
WordEval: Evaluating Word-Native Operation Fidelity in Document Editing
Jinyoung Park, Yupan Huang, Shimin Zhang, Jiayu Ding, Yilin Jia, Yuzhong Zhao, Tengchao Lv, Wenshan Wu, Shaohan Huang, Nan Yang, Xiangyang Zhou, Dongdong Zhang and 9 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WordEval, an evaluation framework for checking whether agents perform word-native operations (formatting, lists, fields, layout, object properties) correctly when editing Microsoft Word documents.
Problem
Evaluating whether an agent correctly edited a Microsoft Word document is hard, because Word edits change a structured document object (formatting, lists, fields, layout, properties) that text similarity or rendered visual comparison protocols fail to capture.
Approach
A word-native evaluation protocol that checks the structured document object for the requested operation effects, rather than whole-document text similarity or rendered visual comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Move from single atomic edits to combined compositional edits | Claude Code (Opus 5) · WordEval | case-level pass rate (%): 87.5 -> 84.4 | Hurt |
| Move from atomic to compositional edits | ZCode (GLM-5.2) · WordEval | case-level pass rate (%): 58.2 -> 39.6 | Hurt |
| Move from atomic to compositional edits | OpenCode (Gemini 3.8 Flash) · WordEval | case-level pass rate (%): 57.9 -> 49.0 | Hurt |
| Check object properties with Word-native oracle instead of SSIM >= 0.99 rendering comparison | evaluator · Evaluator validation | false-positive rate (%): 45.1 -> 5.5 | Improved |
| Use Word Object oracle instead of Word Revision diff | evaluator · Evaluator validation | false-negative rate (%): 59.9 -> 16.8 | Improved |
| Raise SSIM threshold from 0.99 to 0.999 | SSIM evaluator · Evaluator validation | FPR / FNR (%): FPR 45.1 -> 7.7; FNR 16.2 -> 27.5 | Mixed |
| Condition compositional pass rate on all linked atomic variants passing (additive tasks) | Kimi Code (K3) and Codex (GPT-5.6) · WordEval Compositional additive | case pass rate: +41.6 pts for K3; +2.3 pts for GPT-5.6 | Mixed |
| Condition on atomic success for layout-coupled tasks | Codex (GPT-5.6) · WordEval layout-coupled | case pass rate (%): 76.9 -> 66.7 | Hurt |
Experimental setup
Benchmarks: WordEval
Baselines: whole-document text similarity protocols rendered visual comparison protocols
Closest papers by meaning
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.83)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.83)
- Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration (AAAI 2026 · similarity 0.83)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.82)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.82)
- MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
