Research map / Large language models / Language, safety and interpretability

GiLOT: Interpreting Generative Language Models via Optimal Transport

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GILOT, a perturbation-based input attribution method for generative LLMs that uses optimal transport with token-embedding cost to measure output distribution shifts when each input token is masked, evaluated with MoRF/LeRF/ABPC faithfulness on Llama-family models.

Problem

Existing feature attribution methods designed for discriminative models like BERT yield unfaithful explanations for generative LLMs, because output distributions over large vocabularies have unequal semantic distances and outputs are autoregressive sequences.

Approach

Measure the change in next-token distributions (marginalized over beam-searched continuations) when an input token is masked, using optimal transport with a cosine-distance cost between token embeddings instead of KL divergence, so that semantically similar token shifts are penalized less.

Key results

• On Llama2-13b-chat, GILOT achieves ABPC 1.38 vs 0.93 for KLD variant and ranks highest among baselines (LIME, IG, BTA) on most faithfulness metrics across Llama2, Alpaca-LoRA, and Vicuna models.
• Attribution-guided query weight correlates with SST-2 accuracy under example permutation (Pearson 0.7697, p≈1e-11) and negatively with order sensitivity (-0.7688).
• Increasing beam size from 2 to 10 raises ABPC from 1.12 to 1.38 (23.2% gain) on Llama2-13b-chat.

Measured effects reported in the paper

Surprising findings

• Masking the token 'protagon-' (leaving '-ist') was flagged as important by GILOT, and the paper notes that remaining suffix can change the model's output drastically.
• Simple IG-Avg and BTA-Avg variants sometimes outperform GILOT on individual metrics (e.g., IG on MoRF for Vicuna-7B), so GILOT does not dominate across all settings.

Limitations

• Computation requires an OT solve per input token, which becomes unmanageable for very long contexts (up to 128K tokens).
• Beam-search approximation of the output distribution can be inaccurate when the model generates highly diverse outputs.
• Evaluation relies on re-generated response embedding discrepancy, a proxy metric; the authors acknowledge no established metric for generative faithfulness.

Experimental setup

Datasets: Alpaca ShareGPT SST-2

Benchmarks: MoRF/LeRF/ABPC perturbation tests on 100 Alpaca and ShareGPT prompts SST-2 in-context order permutation analysis

Metrics: MoRF LeRF ABPC Pearson correlation accuracy order sensitivity

Baselines: LIME IG IG-Avg BTA BTA-Avg KLD-based estimator

Builds on: LIME Integrated Gradients Bidirectional Transformer Attribution Sinkhorn-Knopp optimal transport IPOT

Closest papers by meaning

• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.85)
• AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution (ICLR 2025 · similarity 0.85)
• Large Language Diffusion Models (NeurIPS 2025 · similarity 0.85)
• Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs (ICML 2026 · similarity 0.85)
• Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.84)
• Probabilistic Token Alignment for Large Language Model Fusion (NeurIPS 2025 · similarity 0.84)