Research map / Image generation / Editing and style transfer
NEP: Autoregressive Image Editing via Next Editing Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Next Editing-token Prediction (NEP), an autoregressive image editing method that regenerates only masked editing-region tokens, built on a randomized-order LlamaGen (RLlamaGen) pre-trained for any-order generation, plus a Grad-CAM-guided test-time self-refinement loop.
Problem
Text-guided image editing typically regenerates the entire target image, wasting computation and biasing training toward reconstructing non-edited regions rather than the intended edits.
Approach
Formulate editing as predicting only the tokens inside user-specified editing regions, using an any-order autoregressive T2I model (RLlamaGen) pre-trained with learned shuffled positional embeddings, and add editing/non-editing mask embeddings to condition generation; test-time scaling iteratively regenerates low-scoring regions chosen by Grad-CAM over CLIP tokens and ImageReward.
Key results
- On MagicBrush single-turn editing, NEP achieves the best L1 (0.0547), L2 (0.0163), CLIP-I (0.9350) and DINO (0.9044) among compared methods.
- On MS-COCO 30K, test-time scaling with NEP improves RLlamaGen FID from 11.49 to 9.82 (4 revision rounds) with CLIP rising from 0.325 to 0.332.
- NEP inference time is 2.88 s versus 2.94 s for UltraEdit and 10.70 s for EditAR, but peak memory is higher at 13.25 GB versus 4.04 GB for UltraEdit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NEP rather than full-image NTP generation (ablation) | NEP image editing · MagicBrush multi-turn | L1: 0.0968 -> 0.0712 | Helped |
| Add editing and unediting mask embeddings (ERC) (ablation) | NEP image editing · MagicBrush multi-turn | L1: 0.0741 -> 0.0712 | Helped |
| Use random in-mask order instead of in-mask raster scan (ablation) | RLlamaGen zero-shot editing · MagicBrush single-turn | L1: 0.0743 -> 0.0741 | No clear effect |
| Use random in-mask order instead of in-mask raster scan (ablation) | RLlamaGen zero-shot editing · MagicBrush multi-turn | L1: 0.0916 -> 0.0911 | No clear effect |
| Add NEP test-time scaling to RLlamaGen | RLlamaGen text-to-image generation · MS-COCO 30K | CLIP: 0.325 -> 0.330 | Helped |
| Add NEP test-time scaling to RLlamaGen | RLlamaGen text-to-image generation · MS-COCO 30K | FID: 11.49 -> 10.18 | Helped |
| Add one revision round (ablation) | RLlamaGen with NEP test-time scaling · MS-COCO 30K | FID: 11.49 -> 9.94 | Helped |
Surprising findings
- Removing editing region conditioning makes the model more likely to leave the source image unchanged.
- Random generation order within the mask gives negligible difference from raster order for zero-shot editing.
Limitations
- Requires user-provided editing masks, adding annotation or computation cost.
- Robustness to masks that are smaller than the true edit region is not addressed.
- Higher memory consumption due to mask token concatenation.
Experimental setup
Datasets: ALLaVA-LAION CC12M Kosmos-G LAION-LVIS-220 LAION-COCO-AESTHETIC LAION-COCO-17M ShareGPT4V UltraEdit
Benchmarks: MagicBrush Emu Edit MS-COCO 30K
Metrics: L1 L2 CLIP-I DINO CLIP-T CLIPdir CLIPout CLIPimg FID CLIP score
Baselines: UltraEdit EditAR FireEdit AnySD InstructPix2Pix MagicBrush HIVE Null Text Inversion GLIDE Blended Diffusion SD-SDEdit MIGE
Builds on: LlamaGen MaskGIT UltraEdit ImageReward
Closest papers by meaning
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.90)
- The Promise of RL for Autoregressive Image Editing (NeurIPS 2025 · similarity 0.89)
- Edit the Bits, Diff the Codes: Bitwise Residual Editing for Visual Autoregressive Models (NeurIPS 2026 · similarity 0.87)
- AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing (AAAI 2024 · similarity 0.87)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.86)
- EditInfinity: Image Editing with Binary-Quantized Generative Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.