Research map / Large language models / Language, safety and interpretability
LLMs Process Lists With General Filter Heads
Arnab Sen Sharma, Giordano Rogers, Natalie Shapira, David Bau
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| ablate filter heads (attention restricted to BOS) (ablation) | Llama-70B · Select* tasks | task accuracy | Hurt |
| ablate filter heads (ablation) | Llama-70B · Counting and CheckExistence | task accuracy | No clear effect |
| ablate is_match flag in item latents with question placed before items (ablation) | Llama-70B · SelectOne objects | accuracy: 46.09% with ablation (question before) | Hurt |
| ablate is_match flag with question placed after items (ablation) | Llama-70B · SelectOne objects | accuracy: 96.06% with ablation (question after) | No clear effect |
| place question before the items instead of after | filter heads · SelectOne objects | causality: near zero | Hurt |
| add priming prefixes recalling relevant information | filter heads · person-nationality and landmark-country | causality: approximately 10 points | Improved |
| transfer filter heads to non-semantic filtering (rhyme) | filter heads · rhyming words | causality | Hurt |
| increase distractors in collection up to 7 | filter heads · SelectOne objects | causality: close to 0.8 | No clear effect |
| use learned sparse masks over heads instead of single-head patching (ablation) | filter head localization · SelectOne objects | collective causality | Improved |
| swap key states of target and other item in addition to query patch (ablation) | filter heads · SelectOne objects | causality: 0.783 (432/552) | Improved |
| transfer filter heads across Select* tasks | filter heads · SelectOne, MCQ, First, Last | causality: at least 70% | Improved |
| transfer Select* filter heads to Counting | filter heads · Counting | causality | Hurt |
Closest papers by meaning
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.85)
- HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis (NeurIPS 2024 · similarity 0.85)
- Heads up! Large Language Models Can Perform Tasks Without Your Instruction via Selective Attention Head Masking (ICML 2025 · similarity 0.84)
- Thought of Search: Planning with Language Models Through The Lens of Efficiency (NeurIPS 2024 · similarity 0.84)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.84)
- (How) Do Language Models Track State? (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
