Research map / Multi-agent systems and games / LLM agents
Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
Zhilin Liu, Ye Huang, TingXie TingXie, Ruizhi Zhang, Wen Li, Lixin Duan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VF-Coder visual-feedback debugging | Gemini-3-Flash · InteractGUI Bench (PySide6), Gemini-3-Flash base | % Resolved: 21.68 -> 28.29 | Improved |
| Add Gemini CLI text-feedback agent | Gemini-3-Flash · InteractGUI Bench (PySide6), Gemini-3-Flash base | % Resolved: 21.68 -> 24.81 | Improved |
| Add Cursor CLI text-feedback agent | Gemini-3-Flash · InteractGUI Bench (PySide6), Gemini-3-Flash base | Avg. Visual Score: 0.4284 -> 0.5028 | Improved |
| Add GUI Operator (visual interaction debugging) (ablation) | VF-Coder · InteractGUI Bench ablation (Gemini-3-Flash) | % Resolved: 24.79 -> 27.64 | Improved |
| Give Code Fixer the bug screenshot (ablation) | VF-Coder · InteractGUI Bench ablation (Gemini-3-Flash) | % Resolved: 27.64 -> 28.29 | Improved |
| Give Code Fixer the bug screenshot (ablation) | VF-Coder · InteractGUI Bench ablation (Gemini-3-Flash) | Avg. Visual Score: 0.5390 -> 0.5584 | Improved |
| VF-Coder vs Kimi Agent on 80-app subset | Kimi-K2.5 · InteractGUI Bench subset (80 apps, manual), Kimi-K2.5 base | % Resolved: 18.75 -> 25.00 | Improved |
| VF-Coder vs Kimi Agent on visual score | Kimi-K2.5 · InteractGUI Bench subset (80 apps, manual), Kimi-K2.5 base | Avg. Visual Score: 0.5648 -> 0.5530 | Hurt |
| Add VF-Coder on top of Bolt.diy | Bolt.diy · WebGen-Bench | Accuracy (%): 31.8 -> 41.8 | Improved |
| Add VF-Coder on Design2Code-Hard text-heavy dimensions | Gemini-3-Flash · Design2Code-Hard, Gemini-3-Flash base | Text: Self-Revision 98.65 vs VF-Coder 96.31 | Mixed |
| Add VF-Coder vs Cursor CLI on cost | Gemini-3-Flash · InteractGUI Bench (PySide6), Gemini-3-Flash base | Avg. cost ($): 0.1726 -> 0.2064 | Hurt |
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
- JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.87)
- Code2World: A GUI World Model via Renderable Code Generation (NeurIPS 2026 · similarity 0.87)
- Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.86)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.86)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
