Research map / Large language models / Language, safety and interpretability
AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttriBoT, a bundle of efficiency techniques (KV caching, hierarchical attribution, proxy-model modeling and pruning) that approximates leave-one-out context attribution for LLMs, achieving over 300x speedup while staying faithful to the target model's LOO scores.
Problem
Leave-one-out (LOO) context attribution for LLMs requires one forward pass per context source, making it prohibitively expensive for large models and long contexts.
Approach
Combine lossless KV caching of shared prefixes, hierarchical group-then-source attribution, and small same-family proxy models (with optional proxy-based pruning) to approximate the 70B/72B target model's LOO attributions at a fraction of the cost, with the components composable into a Pareto-optimal efficiency-faithfulness trade-off.
Key results
- AttriBoT achieves over 300x speedup over exact LOO attribution while remaining more faithful to the target model's LOO error than prior context attribution baselines such as ContextCite.
- Paragraph-level LOO scores correlate with the sum of sentence-level LOO scores at Pearson R = 0.97 on HotpotQA with Llama 70B.
- Proxy model LOO attributions correlate with Llama 70B at Pearson 0.94 for the 8B proxy and 0.88 for the 1B proxy.
- On HotpotQA against human annotations, AttriBoT achieves a 300x speedup with only about a 10% drop in mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse cached key-value activations during LOO scoring | LOO attribution with Llama 3.1 70B Instruct · HotpotQA | Speedup: 1.6× | Helped |
| Reuse cached key-value activations during LOO scoring | LOO attribution · HotpotQA | Outlier recovery | Helped |
| Use a smaller same-family proxy model to score the target response | Llama 3.1 70B Instruct LOO attribution · HotpotQA | Pearson correlation: 0.94 | Helped |
| Use the smaller 1B same-family proxy model instead of the 8B proxy | Llama 3.1 70B Instruct LOO attribution · HotpotQA | Pearson correlation: 0.94 -> 0.88 | Hurt |
| Score paragraphs first, then sentence-level spans only in retained paragraphs | Hierarchical attribution · HotpotQA | Pearson correlation of paragraph score and summed sentence scores: R = 0.97 | Helped |
| Use a proxy model to prune low-attribution sources before target-model rescoring | Proxy model pruning with Llama 8B proxy · HotpotQA | Outlier recall: 85% | Helped |
| Combine the proposed acceleration techniques | AttriBoT · Open-book QA (three datasets) | Speedup: >300× | Helped |
| Use AttriBoT rather than exact LOO for human-annotated source attribution | AttriBoT · HotpotQA | mAP: 10% drop | Hurt |
| Use a proxy model from a mismatched model family | AttriBoT proxy methods · SQuAD, HotpotQA, and QASPER | Attribution faithfulness | Hurt |
Surprising findings
- Attention weights baseline is surprisingly competitive, reaching mAP about 10 points higher with Qwen 72B than with Llama 70B and nearly Pareto-optimal.
- AttriBoT's methods track human annotations more closely than they track the target model's own LOO attributions.
Limitations
- Attribution fidelity drops for smaller or mismatched-family proxy models (e.g., Mistral 7B proxy degrades performance).
- Evaluation is limited to open-book QA on three datasets, with the authors noting other tasks are untested.
- Duplicated input (common in RAG) can break LOO attribution, requiring deduplication preprocessing.
Experimental setup
Datasets: SQuAD 2.0 HotpotQA QASPER
Benchmarks: SQuAD 2.0 HotpotQA QASPER
Metrics: mean average precision GPU-seconds FLOPs Pearson correlation outlier recall
Baselines: ContextCite Attention Weights Gradient Norm Sentence embedding similarity LOO (exact)
Builds on: ContextCite Generalized ESD test
Closest papers by meaning
- DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
- WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models (NeurIPS 2024 · similarity 0.87)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.87)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.87)
- ContextCite: Attributing Model Generation to Context (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.