Research map / Large language models / Reasoning and chain of thought

DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DecodeShare, which finds a low-dimensional cross-task subspace in KV-cached decode-time hidden states and tests its causal role by projecting it out only during decoding, with energy-matched controls and patchback.

Problem

Unclear whether LLMs use task-general structure at decode time (KV-cached single-token steps) rather than during prefill, and whether activation-steering directions are reliable when estimated from prefill activations.

Approach

Pool task-centered decode-time hidden states, run PCA, select components shared across tasks by a relative variance-share threshold, and causally test them via decode-only projection removal against dimension- and energy-matched controls; also compare prefill- versus decode-estimated bases and use decode-time validation to select steering vectors.

Key results

• On Llama-2-7B-Chat layer 10, removing the 32D shared decode subspace drops forced-choice accuracy from 44.1% to 28.5% (-15.6 points), while removing a 3D readout slice leaves accuracy unchanged (+0.8).
• Patching back only the shared component restores the ablated answer on 100% of flips at full window across the 9 model-layer settings in the patchback table, while random-vector and random-subspace controls rescue at most about 37-50%.
• Decode-aligned validation selects steering vectors with mean REAL accuracy gain +0.011 and flip rate 0.083, versus -0.002 and 0.750 for prefill-aligned validation.

Surprising findings

• Prefill-estimated subspaces largely fail to reproduce the decode-time causal effect, with ablation behaving like matched random controls.
• Projecting out the shared component from steering vectors can trade off average effect against robustness, with the magnitude of robustness gains being task-dependent.

Limitations

• Requires white-box access and per-model estimation, so not applicable to black-box models.
• Shared-set selection depends on thresholds tau and retention rho; the regime is shown empirically rather than derived.
• Results reported mainly for 7B-scale models on multiple-choice benchmarks; scale checks are described only briefly.

Experimental setup

Datasets: GSM8K AQuA CommonsenseQA PIQA OpenBookQA QASC BoolQ StrategyQA ARC-Challenge LogiQA HumanEval SST-2

Benchmarks: GSM8K AQuA CommonsenseQA PIQA OpenBookQA QASC BoolQ StrategyQA ARC-Challenge LogiQA HumanEval SST-2

Metrics: forced-choice accuracy exact match rescue rate flip rate regret@1 paired bootstrap CI sign-flip test

Baselines: prefill-estimated subspace random subspace energy-matched non-shared control CAA contrastive vectors SAE features

Builds on: activation patching representation engineering contrastive activation addition

Closest papers by meaning

• Distribution-Aligned Decoding for Efficient LLM Task Adaptation (NeurIPS 2025 · similarity 0.87)
• DISCO: Disentangled Communication Steering for Large Language Models (NeurIPS 2025 · similarity 0.86)
• When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search (NeurIPS 2026 · similarity 0.86)
• Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding (ICML 2026 · similarity 0.85)
• The Sharp Directions Are Against You: Curvature Analysis of Activation Steering (NeurIPS 2026 · similarity 0.85)
• Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs (NeurIPS 2026 · similarity 0.85)