Research map / Large language models / Instruction and fine-tuning

In-Context Editing: Learning Knowledge from Self-Induced Distributions

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Consistent In-Context Editing (ICE), which fine-tunes an LLM toward a self-generated in-context distribution instead of a one-hot target, to improve knowledge editing accuracy, locality, generalization and fluency.

Problem

Gradient-based fine-tuning for knowledge editing targets one-hot answers, which causes overfitting, poor locality, and unnatural generation, especially under continual edits.

Approach

Tune the model so its output distribution conditioned on the query matches its own in-context distribution (conditioned on a GPT-4-generated context), with the in-context target refreshed each optimization step and samples drawn from the context-plus-target input; this turns the one-hot target into a dynamic self-induced distribution.

Key results

• On Llama2-7b-chat ZsRE, ICE reaches 100.00 edit success, 55.52 portability, 56.97 locality and PPL_r 15.50, versus FT-M's 54.47 portability and 53.84 locality.
• On WikiData_counterfact with Llama2-7b-chat, ICE gets 100.00 edit success, 73.49 portability, 45.88 locality and PPL_r 18.95 (FT-M: 72.39 / 40.76 / 54.71).
• In continual editing on Wiki_recent with Llama2-7b-chat, ICE attains 100.00 edit success, 59.27 portability, 38.33 locality and PPL_r 0.10.

Measured effects reported in the paper

Surprising findings

• Sampling without the in-context prompt (self-distillation without context) gives the worst results, and the static-target variant of ICE is equivalent to combining sampling with fine-tuning, which the authors show is equivalent to plain fine-tuning under their proof.
• Gains over FT-M are smaller on GPT2-xl than on Llama2-7b-chat, which the authors attribute to weaker in-context learning in GPT2-xl.

Limitations

• Does not use the model's internal representations; only contextual prompts and output distributions are exploited.
• Gains are smaller for weaker in-context learners such as GPT2-xl.
• Editing relies on GPT-4-generated contexts, and locality and portability trade off against accuracy.

Experimental setup

Datasets: WikiData_recent ZsRE WikiBio WikiData_counterfact

Benchmarks: WikiData_recent ZsRE WikiBio WikiData_counterfact

Metrics: edit success portability locality fluency PPL_r

Baselines: ROME MEMIT FT-L FT-M

Builds on: ROME MEMIT fine-tuning with constraints (projected gradient descent) KnowEdit

Closest papers by meaning

• Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.91)
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.89)
• Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis (NeurIPS 2026 · similarity 0.89)
• UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Revealing and Mitigating Over-Attention in Knowledge Editing (ICLR 2025 · similarity 0.89)
• Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.89)