Research map / Architectures and efficiency / Attention and transformers

DePass: Unified Feature Attributing by Simple Decomposed Forward Pass

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Transformer block and depth design · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DePass, a single decomposed forward pass that splits hidden states into additive components and propagates them with attention scores and MLP activations frozen, enabling exact token-, component-, and subspace-level attribution without auxiliary training.

Problem

Attributing Transformer behavior to internal computations in a faithful, fine-grained way; existing gradient, attention, ablation, and approximation methods are either unfaithful, costly, or fail to reach neuron/head granularity.

Approach

Decompose each token's hidden state into additive components (per input token, per attention head/residual, per MLP neuron, or per subspace) and propagate them exactly through the network by freezing attention scores and MLP activations, with MLP activation contributions apportioned to components via a softmax over component relevance scores. Per-component contributions sum exactly to the original

Key results

• On Known_1000 and IOI with Llama-2-13b-chat-hf, DePass yields substantially higher comprehensiveness than gradient and attention baselines, while sufficiency is comparable across methods.
• Using DePass-based masking of 30% of misleading tokens, Llama-2-7b-chat-hf CounterFact accuracy rises from 10.16% (misinformation) to 43.13%, versus 25.68% with TACS masking.
• Softmax-based MLP apportioning outperforms linear-norm and linear-weighted alternatives on patch-top comprehensiveness (e.g., 80.8% vs 64.8% and 59.4% at 0.1 patch level).

Measured effects reported in the paper

Surprising findings

• Attention-based methods preserve broad semantic structure and perform slightly better on sufficiency than DePass.
• On TruthfulQA, TACS masking outperforms DePass masking for Qwen2-1.5B-Instruct and Qwen2-7B-Instruct.

Limitations

• Softmax apportioning of MLP activations is heuristic, as acknowledged by the authors.
• Sufficiency results are comparable to baselines, and evaluation relies on token masking which can be limited.
• Evaluated mainly on chat-tuned Llama and Qwen models of up to 70B; generality to other architectures is only claimed.

Experimental setup

Datasets: Known_1000 IOI CounterFact TruthfulQA

Benchmarks: Known_1000 IOI CounterFact TruthfulQA

Metrics: comprehensiveness sufficiency accuracy relative change in predicted probability

Baselines: Input x Gradient Integrated Gradients Gradient SHAP Attention Rollout Last-layer Attention Mean Attention AtP TACS

Builds on: Integrated Gradients AtP DecompX Attention Rollout TACS

Closest papers by meaning

• Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.87)
• Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers (CVPR 2025 · similarity 0.85)
• Interpreting vision transformers via residual replacement model (NeurIPS 2025 · similarity 0.85)
• Observable Propagation: Uncovering Feature Vectors in Transformers (ICML 2024 · similarity 0.85)
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability (NeurIPS 2025 · similarity 0.85)
• Unlearning-based Neural Interpretations (ICLR 2025 · similarity 0.85)