Research map / Large language models / Language, safety and interpretability
PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaceLLM, a training-free (plus low-cost fine-tuning) brain-inspired method combining an activation-level memory bank (persistent activity) and balanced clustering of FFN neurons into cortical experts to improve long-context understanding.
Problem
LLM long-context performance is limited by transient activations causing information decay and unstructured FFN weights causing semantic fragmentation.
Approach
Adds an Activation Memory Bank in FFN layers that retrieves, fuses and updates cached FFN activations by cosine similarity (persistent activity), and reorders FFN weights via constrained KMeans clustering of gated-projection rows into expert blocks (cortical experts), without retraining.
Key results
- Activation Beacon + CE + PA reaches 46.55 on LongBench Multi-document QA vs 40.30 for Activation Beacon (about 6 points gain).
- On ∞-Bench, the method improves En.Dialogue from 3.00 to 15.5 and En.MC from 46.72 to 64.19 over Activation Beacon.
- Training-free Qwen-2-7B-Instruct with CE+PA improves LongBench Multi-document QA average from 49.03 to 50.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CE and PA | Qwen-2-7B-Instruct · LongBench MQA, training-free | score: 49.03 -> 50.28 | Helped |
| Add CE and PA | Qwen-2-7B-Instruct · LongBench Cod., training-free | score: 50.05 -> 49.95 | Hurt |
| Add CE and PA | Llama-2-7B-chat · LongBench SQA, training-free | score: 23.92 -> 25.35 | Helped |
| Add CE and PA to Activation Beacon | Qwen-2-7B-Instruct · LongBench MQA, low-cost fine-tuning | score: 40.30 -> 46.55 | Helped |
| Apply PaceLLM instead of Activation Beacon | Long-context benchmark model · ∞-Bench En.Dia | score: 3.00 -> 15.5 | Helped |
| Apply PaceLLM instead of Activation Beacon | Long-context benchmark model · ∞-Bench En.MC | score: 46.72 -> 64.19 | Helped |
| Apply PaceLLM instead of Activation Beacon | MMLU model · MMLU Avg. | score: 70.250 -> 70.510 | Helped |
| Add CE (ablation) | Qwen-2-7B-Instruct · LongBench SQA, training-free | score: 37.76 -> 37.68 | Hurt |
| Add PA (ablation) | Qwen-2-7B-Instruct · LongBench SQA, training-free | score: 37.76 -> 38.09 | Helped |
| Add CE (ablation) | Qwen-2-7B-Instruct · LongBench MQA, training-free | score: 49.03 -> 48.80 | Hurt |
| Add PA (ablation) | Qwen-2-7B-Instruct · LongBench MQA, training-free | score: 49.03 -> 49.36 | Helped |
| Remove noise addition (ablation) | PaceLLM AMB · LongBench SQA, Qwen2-7B | score: 41.62 -> 40.70 | Hurt |
Surprising findings
- Deploying the method at all layers consistently underperforms sparse deployment, and the best layer choice varies by task.
Limitations
- The activation memory bank adds extra computation and storage cost.
- Method applied only to text; multimodal and embodied extensions are untested.
- Hyperparameters (thresholds, layers) vary by task and are tuned per setting.
Experimental setup
Datasets: LongBench ∞-Bench NIAH MMLU
Benchmarks: LongBench ∞-Bench Needle-In-A-Haystack MMLU
Metrics: accuracy score retrieval accuracy
Baselines: Vanilla Activation Beacon LongLLMLingua SnapKV Mistral-7B-Instruct-v0.3
Builds on: Activation Beacon CAMELoT LongLLMLingua SnapKV
Closest papers by meaning
- Human-inspired Episodic Memory for Infinite Context LLMs (ICLR 2025 · similarity 0.88)
- SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking (ICLR 2025 · similarity 0.88)
- Larimar: Large Language Models with Episodic Memory Control (ICML 2024 · similarity 0.88)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.88)
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.88)
- Scaling and context steer LLMs along the same computational path as the human brain (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.