Research map / Architectures and efficiency / Attention and transformers
Causal Edit Serialization
Louis Robinson
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Causal Edit Serialization (CES), a decoder-only transformer extension that emits explicit edit programs in a serialized DSL (insert, delete, replace, move, copy) under a standard causal mask.
Problem
Editing existing sequences with autoregressive decoders typically requires regenerating whole outputs; the paper targets explicit, localized edit generation within a causal decoder.
Approach
Serialize edits as an explicit DSL emitted by a decoder-only transformer while keeping the standard causal mask, augmented with a lightweight localization head and dual-index rotary position embeddings that separate generation and source positions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CES edit serialization instead of clean autoregressive pretraining | NanoChat depth 24 · ClimbMix validation | bits per byte: 0.7484 -> 0.7629 | Hurt |
| use CES addressing (pointer head and dual-index RoPE) instead of AR-DSL numeric or flat-relative addresses (ablation) | NanoChat pretrained edit model · Controlled ClimbMix repair | valid programs: 90.53 valid vs none valid | Improved |
| use CES instead of iterative Levenshtein for partial repair | NanoChat depth 24 · Controlled ClimbMix repair | inputs closer to target / mean similarity gain: 53.40 vs 39.03 closer; 0.00941 vs 0.01062 gain | Mixed |
| supply gold edit locations (and gold operation envelope) to CES (ablation) | CES depth 24 seed 42 · Controlled ClimbMix repair | exact recovery (%): 5.27 -> 7.3 (gold pointers), 8.4 (plus envelope) | No clear effect |
| share one pointer token across roles (ablation) | CES depth 12 · Synthetic pointer accuracy | pointer accuracy (%): 48.64 -> 45.81 | Hurt |
| raise position-loss weight from 0.005 to 0.01 (ablation) | CES depth 12 · Controlled repair validity | validity: +8.27 points validity, +7.07 points similarity-improvement rate, 0.16% BPB change | Improved |
| train on GPT-5 Mini-only edits instead of mixed random plus GPT-5 Mini data (ablation) | CES depth 18 · Controlled repair | valid nontrivial-edit rate: 50.93 (mixed) vs 90.40 unchanged output (GPT-only) | Hurt |
| train on random-only edits instead of mixed data (ablation) | CES depth 18 · Controlled repair | CORE and nontrivial-edit rate vs localization/exact recovery | Mixed |
| add conservative acceptance filter at tau=0.5 (ablation) | post-trained CES d24 · CoEdIT confirmation, GPT-5 Mini judge | net judged improvement (points): +4.74 -> +9.26 | Improved |
| make editing more aggressive (tau=-1) (ablation) | post-trained CES d24 · CoEdIT confirmation, GPT-5 Mini judge | net judged improvement (points): +4.74 -> -4.38 | Hurt |
| append a correct repair program before continuation | CES · 540 held-out ClimbMix documents | NLL of next 128 tokens (nats/token): -0.02123 vs empty program; -0.00641 vs sham program | Improved |
| sample at T=1.0 instead of greedy decoding (ablation) | CES depth 24 · Controlled ClimbMix repair | similarity gain dS: +0.00877 -> -0.00121 | Hurt |
Closest papers by meaning
- Rethinking Residual Distribution in Locate-then-Edit Model Editing (NeurIPS 2025 · similarity 0.86)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.86)
- Resolving Lexical Bias in Model Editing (ICML 2025 · similarity 0.86)
- Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024 · similarity 0.85)
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.85)
- MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-off (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
