Research map / Architectures and efficiency / Pruning and efficient tuning

On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Sparse training from scratch · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits parsimoniously activated dictionary learning (PADL), gives a MAP interpretation via Beta priors and an auxiliary activation latent, derives tradeoff bounds and a threshold rule, and applies it to image reconstruction and VLM token compression.

Problem

Structured dictionary learning with global atom-activation constraints lacks a tractable generative view, and its hyperparameters (lambda1, lambda2, activation threshold) are chosen by grid search, hindering the sparsity-storage-accuracy tradeoff analysis.

Approach

Shows that an l1 plus row-wise l_infinity penalized dictionary objective is the MAP estimate of a model with Beta coefficient priors and an auxiliary atom-activation latent (sigmoid of max coefficient minus threshold), enabling closed-form expected active-set size, high-probability error bounds, and a data-driven threshold rule.

Key results

• On CIFAR-100 reconstruction, PADL reaches RMSE 0.099 and PSNR 20.56 versus 0.106 / 20.13 for CRsAE and 0.112 / 18.98 for L1-regularized DL, using 19 activated atoms.
• On image VQA with LLaVA-1.5, PADL+ (8x) scores 78.2 on VQAv2 and 68.8 on ScienceQA, versus 78.5 and 66.8 for the uncompressed LLaVA-1.5.
• On video QA with Video-LLaVA, PADL+ (16x) reaches 75.2 on MSVD-QA and 64.1 on MSRVTT-QA, above the Video-LLaVA baseline of 70.7 and 59.2.

Surprising findings

• Plain PADL (no adapter fine-tuning) scores below the uncompressed LLaVA-1.5 on VQAv2 (76.9 vs 78.5) and ScienceQA-I (65.5 vs 66.8).

Limitations

• Theoretical bounds rely on approximations (Taylor step for Beta log-prior, restricted active-span condition, Hoeffding proxy) and a proxy objective that treats alpha_S as constant.
• The VLM results do not consistently beat baselines; plain PADL trails the uncompressed model on several benchmarks.
• Experiments are on small grayscale 8x8 patches for the main validation, with a fixed dictionary size of 128 atoms.

Experimental setup

Datasets: CIFAR-100 SVHN LLaVA-1.5 instruction-tuning dataset Valley

Benchmarks: CIFAR-100 SVHN VQAv2 ScienceQA TextVQA POPE MME MMBench MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA

Metrics: RMSE PSNR SSIM accuracy GPT-based score

Baselines: KSVD DDL GDDL BPFA IBP CRsAE JumpReLU SAE L1-regularized DL LLaVA-PruMerge FreePruner

Builds on: K-SVD sparse coding BPFA Indian Buffet Process dictionary learning LLaVA-1.5 Video-LLaVA

Closest papers by meaning

• Understanding l4-based Dictionary Learning: Interpretation, Stability, and Robustness (ICLR 2020 · similarity 0.86)
• Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning (ICML 2026 · similarity 0.86)
• Convolutional dictionary learning based auto-encoders for natural exponential-family distributions (ICML 2020 · similarity 0.86)
• Orthogonality-Promoting Dictionary Learning via Bayesian Inference (AAAI 2019 · similarity 0.86)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.85)
• Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models (ICML 2025 · similarity 0.85)