AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Breaking Information Islands in Sparse Tuning via Small-World Connectivity

Xianchao Guan, Zijun Xiong, Yifeng Wang, Yubo Cui, Yaowei Wang, Guosen Xie, Xin Li, Zheng Zhang

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a sparse-tuning approach for LLM adaptation that uses small-world connectivity to avoid information islands among feature dimensions.

Problem

Sparse tuning of large language models depends on the feature-interaction topology induced by the sparse update; limited or uneven cross-dimensional interactions can isolate feature dimensions into information islands that hinder task-specific information mixing.

Approach

Structures the sparse update pattern with small-world connectivity so that feature dimensions are well interconnected, breaking information islands while keeping the trainable parameter count sparse.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add fixed scattering permutation to block-diagonal update (SWIFT vs DiaBlo)block-diagonal sparse tuning · Commonsense reasoning, LLaMA3-8Baverage accuracy (%): 87.00 -> 88.05Improved
add fixed scattering permutation to block-diagonal update (SWIFT vs DiaBlo)block-diagonal sparse tuning · NLG, LLaMA2-7Baverage score (%): 31.53 -> 32.56Improved
add fixed scattering permutation to block-diagonal update (SWIFT vs DiaBlo)block-diagonal sparse tuning · Image classification, CLIP ViT-B/32average accuracy (%): 85.07 -> 86.78Improved
use uniform scattering permutation instead of random permutation (ablation)SWIFT · Image classification, CLIP ViT-B/32average accuracy (%): +0.77 percentage pointsImproved
use uniform scattering permutation instead of diagonal cyclic permutation (ablation)SWIFT · Image classification, CLIP ViT-B/32average accuracy (%): +1.71 percentage pointsImproved
increase update density beyond N=64 (N=32, 3.13%) (ablation)SWIFT · Commonsense reasoning, LLaMA3-8Baverage accuracy (%)No clear effect
use N=64 blocks (density 1.56%) as default (ablation)SWIFT · Commonsense reasoning, LLaMA3-8Baverage accuracy (%): 88.05Improved
apply SWIFT on attention projections only vs DiaBlo (ablation)SWIFT · Commonsense reasoning, LLaMA3-8Baverage accuracy (%): +1.24 percentage pointsImproved
apply SWIFT on feed-forward projections only vs DiaBlo (ablation)SWIFT · Commonsense reasoning, LLaMA3-8Baverage accuracy (%): +0.93 percentage pointsImproved
use block-diagonal structure with scattering permutation for MRGCSWIFT support · MRGC, d=4096, K=64MRGC: 0.00 -> 1.00Improved
use block-diagonal permuted support instead of random masksSWIFT support · MRGC, d=4096, K=64MRGC: 0.41 -> 1.00Improved
use SWIFT instead of LoRALLaMA3-8B · Commonsense reasoningaverage accuracy (%): 85.01 -> 88.05Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy