AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

WordEval: Evaluating Word-Native Operation Fidelity in Document Editing

Jinyoung Park, Yupan Huang, Shimin Zhang, Jiayu Ding, Yilin Jia, Yuzhong Zhao, Tengchao Lv, Wenshan Wu, Shaohan Huang, Nan Yang, Xiangyang Zhou, Dongdong Zhang and 9 more

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Text generation and structured prediction · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WordEval, an evaluation framework for checking whether agents perform word-native operations (formatting, lists, fields, layout, object properties) correctly when editing Microsoft Word documents.

Problem

Evaluating whether an agent correctly edited a Microsoft Word document is hard, because Word edits change a structured document object (formatting, lists, fields, layout, properties) that text similarity or rendered visual comparison protocols fail to capture.

Approach

A word-native evaluation protocol that checks the structured document object for the requested operation effects, rather than whole-document text similarity or rendered visual comparison.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Move from single atomic edits to combined compositional editsClaude Code (Opus 5) · WordEvalcase-level pass rate (%): 87.5 -> 84.4Hurt
Move from atomic to compositional editsZCode (GLM-5.2) · WordEvalcase-level pass rate (%): 58.2 -> 39.6Hurt
Move from atomic to compositional editsOpenCode (Gemini 3.8 Flash) · WordEvalcase-level pass rate (%): 57.9 -> 49.0Hurt
Check object properties with Word-native oracle instead of SSIM >= 0.99 rendering comparisonevaluator · Evaluator validationfalse-positive rate (%): 45.1 -> 5.5Improved
Use Word Object oracle instead of Word Revision diffevaluator · Evaluator validationfalse-negative rate (%): 59.9 -> 16.8Improved
Raise SSIM threshold from 0.99 to 0.999SSIM evaluator · Evaluator validationFPR / FNR (%): FPR 45.1 -> 7.7; FNR 16.2 -> 27.5Mixed
Condition compositional pass rate on all linked atomic variants passing (additive tasks)Kimi Code (K3) and Codex (GPT-5.6) · WordEval Compositional additivecase pass rate: +41.6 pts for K3; +2.3 pts for GPT-5.6Mixed
Condition on atomic success for layout-coupled tasksCodex (GPT-5.6) · WordEval layout-coupledcase pass rate (%): 76.9 -> 66.7Hurt

Experimental setup

Benchmarks: WordEval

Baselines: whole-document text similarity protocols rendered visual comparison protocols

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy