Research map / Large language models / Language, safety and interpretability

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows frontier LLMs fail exact copying within context and proposes 2D-RoPE (row/column position IDs from newlines) that yields length generalization on copy tasks and pretraining gains on binary copy at 350M-1.4B scale.

Problem

Frontier LLMs fail to exactly copy input strings well within their context windows; standard positional encodings (RoPE) favor context-matching shortcuts over position-based copying, which does not length-generalize.

Approach

Assign each token a 2D position (row ID from preceding newlines, column ID from offset within line) and apply RoPE separately along both axes, so copying becomes fixed-column retrieval from the previous row; add Auto-2D-RoPE that learns coordinate updates without explicit newlines.

Key results

• Frontier LLMs including GPT-5.5, Gemini 3.1 Pro, and DeepSeek V4 Pro fail on substantial fractions of copy inputs, with accuracy often well below 50% on longer inputs.
• Synthetic 1-layer 2D-RoPE models achieve perfect copying up to about 1000x training length, while standard PEs collapse beyond training length.
• At 1.4B params and 28B tokens, 2D-RoPE reaches 100.0 at 1K/2K and 92.0 at 4K on imbalanced copy versus 26.8 at 2K and 0.0 at 4K for RoPE, with CSR 46.8 vs 45.9.

Surprising findings

• Even a finetuned Qwen3-8B still fails at copying, indicating the copy failure persists after finetuning.
• Multi-layer RoPE Transformers can express position-based copying in theory, yet training biases them toward context-matching shortcuts.

Limitations

• Original 2D-RoPE relies on explicit newline tokens and degenerates toward RoPE without them; Auto-2D-RoPE is only tested in small settings.
• Theoretical guarantees are for one-layer models; why multi-layer RoPE fails to copy is not fully explained.
• Frontier LLM evaluations use API calls with unverifiable internal training, and results are sensitive to prompt templates and thinking settings.

Experimental setup

Datasets: DCLM FineWeb-Edu synthetic binary copy data (imbalanced, recursive-flip, uniform) synthetic Python list conversion data

Benchmarks: binary copy (imbalanced, recursive-flip) Python list conversion repeat-structure copy test ARC-easy ARC-challenge HellaSwag WinoGrande MMLU LAMBADA PIQA BoolQ

Metrics: exact-match copy accuracy average accuracy (common-sense reasoning) validation loss

Baselines: RoPE ALiBi NoPE RNoPE H-RoPE GPT-5.5 Gemini 3.1 Pro DeepSeek V4 Pro Claude Opus 4.7 Qwen3.6 Plus Qwen3-8B

Builds on: RoPE induction heads Qwen3 DCLM

Closest papers by meaning

• Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · similarity 0.88)
• Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.88)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
• What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.85)
• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.84)
• Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.84)