Research map / Architectures and efficiency / Pruning and efficient tuning
GHOST: Unmasking Phantom States in Mamba2 via Grouped Hidden-state Output-aware Selection & Truncation
Michael Menezes, Anastasios Kyrillidis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use joint controllability x observability saliency instead of controllability only (ablation) | GHOST · WikiText-2 (50% sparsity) | perplexity: 14.56 -> 14.23 | Improved |
| use joint saliency instead of observability only (ablation) | GHOST · WikiText-2 (50% sparsity) | perplexity: 16.16 -> 14.23 | Improved |
| use joint saliency instead of controllability only at 90% sparsity (ablation) | GHOST · WikiText-2 (90% sparsity) | perplexity: 2158 -> 25.07 | Improved |
| use joint saliency instead of controllability only at 70% sparsity (ablation) | GHOST · WikiText-2 (70% sparsity) | perplexity: 16.71 -> 16.16 | Improved |
| replace one-shot Taylor scoring with GHOST forward-pass sequential scoring | Mamba2-1.3B · WikiText-2 (70% sparsity) | perplexity: 4255 -> 16.16 | Improved |
| use GHOST instead of Taylor on small models | Mamba2-130M · WikiText-2 (50% sparsity) | perplexity: 1E16 -> 29.06 | Improved |
| use GHOST instead of Taylor on large models | Mamba2-2.7B · WikiText-2 (50% sparsity) | perplexity: 11.50 -> 12.14 | Hurt |
| calibrate on short context (128) and evaluate on 2048 using GHOST vs Taylor | Mamba2-1.3B · WikiText-2 length generalization | perplexity: 1613 -> 14.27 | Improved |
| use GHOST instead of Magnitude pruning | Mamba2-1.3B · WikiText-2 (50% sparsity) | perplexity: 29.34 -> 14.23 | Improved |
| increase number of calibration samples from 8 to 128 (ablation) | GHOST · WikiText-2 (50% sparsity) | perplexity: 14.28 -> 14.23 | No clear effect |
| prune Mamba2 state with GHOST instead of Taylor on Lambada | Mamba2-1.3B · Lambada zero-shot | accuracy: 59.23% (Taylor) vs 14-point drop from dense | Hurt |
| GHOST instead of Taylor on OOD math | Mamba2-1.3B · MMLU Elementary Math | accuracy: 22.49 -> 25.66 | Improved |
Closest papers by meaning
- Demystifying the Token Dynamics of Deep Selective State Space Models (ICLR 2025 · similarity 0.84)
- Layer-Adaptive State Pruning for Deep State Space Models (NeurIPS 2024 · similarity 0.84)
- UnHiPPO: Uncertainty-aware Initialization for State Space Models (ICML 2025 · similarity 0.83)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.83)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.83)
- POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
