AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Causal Edit Serialization

Louis Robinson

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient language transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Causal Edit Serialization (CES), a decoder-only transformer extension that emits explicit edit programs in a serialized DSL (insert, delete, replace, move, copy) under a standard causal mask.

Problem

Editing existing sequences with autoregressive decoders typically requires regenerating whole outputs; the paper targets explicit, localized edit generation within a causal decoder.

Approach

Serialize edits as an explicit DSL emitted by a decoder-only transformer while keeping the standard causal mask, augmented with a lightweight localization head and dual-index rotary position embeddings that separate generation and source positions.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use CES edit serialization instead of clean autoregressive pretrainingNanoChat depth 24 · ClimbMix validationbits per byte: 0.7484 -> 0.7629Hurt
use CES addressing (pointer head and dual-index RoPE) instead of AR-DSL numeric or flat-relative addresses (ablation)NanoChat pretrained edit model · Controlled ClimbMix repairvalid programs: 90.53 valid vs none validImproved
use CES instead of iterative Levenshtein for partial repairNanoChat depth 24 · Controlled ClimbMix repairinputs closer to target / mean similarity gain: 53.40 vs 39.03 closer; 0.00941 vs 0.01062 gainMixed
supply gold edit locations (and gold operation envelope) to CES (ablation)CES depth 24 seed 42 · Controlled ClimbMix repairexact recovery (%): 5.27 -> 7.3 (gold pointers), 8.4 (plus envelope)No clear effect
share one pointer token across roles (ablation)CES depth 12 · Synthetic pointer accuracypointer accuracy (%): 48.64 -> 45.81Hurt
raise position-loss weight from 0.005 to 0.01 (ablation)CES depth 12 · Controlled repair validityvalidity: +8.27 points validity, +7.07 points similarity-improvement rate, 0.16% BPB changeImproved
train on GPT-5 Mini-only edits instead of mixed random plus GPT-5 Mini data (ablation)CES depth 18 · Controlled repairvalid nontrivial-edit rate: 50.93 (mixed) vs 90.40 unchanged output (GPT-only)Hurt
train on random-only edits instead of mixed data (ablation)CES depth 18 · Controlled repairCORE and nontrivial-edit rate vs localization/exact recoveryMixed
add conservative acceptance filter at tau=0.5 (ablation)post-trained CES d24 · CoEdIT confirmation, GPT-5 Mini judgenet judged improvement (points): +4.74 -> +9.26Improved
make editing more aggressive (tau=-1) (ablation)post-trained CES d24 · CoEdIT confirmation, GPT-5 Mini judgenet judged improvement (points): +4.74 -> -4.38Hurt
append a correct repair program before continuationCES · 540 held-out ClimbMix documentsNLL of next 128 tokens (nats/token): -0.02123 vs empty program; -0.00641 vs sham programImproved
sample at T=1.0 instead of greedy decoding (ablation)CES depth 24 · Controlled ClimbMix repairsimilarity gain dS: +0.00877 -> -0.00121Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy