Research map / Architectures and efficiency / Pruning and efficient tuning

SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Sparse training from scratch · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpecDrop, a parameter-free, category-conditioned soft routing schedule with a fixed denominator for multi-branch networks, and shows gains over No-Routing controls only when the partition aligns with training units (vision), with ties on fuzzy language/LoRA partitions.

Problem

Learned routers in modular networks (MoE, multi-branch) often underperform equal-weight No-Routing baselines at matched parameter budgets; the question is whether the bottleneck is the routing algorithm or the alignment between training-signal granularity and category partitions.

Approach

A fixed, parameter-free routing schedule assigns each category a preferred branch with weight p_a and leaks p_i>0 to others, merged via a category-independent fixed denominator S, with a shared expert added after normalization; gains appear only when each training unit carries one clean category tag.

Key results

• Soft SpecDrop reaches 79.23% top-1 on CIFAR-100 (ResNet-110), +16.15 over matched No-Routing (63.08%) and +4.75 over dense ResNet-110 (74.48%).
• On ImageNet-1K BREEDS-46 (ViT-S/16), SpecDrop reaches 79.89% top-1, +6.53 over No-Routing+SE (73.36%) and +3.20 over tuned Soft MoE (76.69%).
• On SlimPajama-6B, SpecDrop's 45.38 PPL trails matched No-Routing+SE (45.28) by 0.10, within seed noise; on SuperNI/LoRA it gains +0.0012 ROUGE-L over No-Routing+SE, which is statistically zero.

Surprising findings

• Given the same category label, masking a dense model's outputs achieves higher accuracy (85.23 on CIFAR-100, 83.65 on ImageNet) than SpecDrop.
• HardCategory (p_i=0) has 100% branch-category alignment yet the worst top-1 on CIFAR-100 (57.67%), below No-Routing.

Limitations

• Experiments are limited to small-scale models (30M LM, 125M replication); no evaluation at >=10B MoE scale.
• The category label is required at inference and is target-derived on vision settings, so gains include the label's value; baselines do not consume it.
• Dense prediction (segmentation, detection, VQA) is untested and the method's alignment condition may not hold there.

Experimental setup

Datasets: CIFAR-100 ImageNet-1K SlimPajama-6B SuperNI BREEDS

Benchmarks: CIFAR-100 ImageNet-1K BREEDS-46 SlimPajama-6B SuperNI

Metrics: top-1 accuracy validation perplexity ROUGE-L F1 branch-category alignment (Align) pruning sensitivity

Baselines: No-Routing No-Routing+SE HardCategory Stochastic Depth Example-Tied Dropout Contextual Dropout Soft MoE ALF top-k router Mod-Squad COMET Switch Hash Layers

Builds on: Mixture-of-experts Soft MoE Stochastic Depth DEMix HydraLoRA

Closest papers by meaning

• Diversity and Depth in Per-Example Routing Models (ICLR 2019 · similarity 0.85)
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (NeurIPS 2020 · similarity 0.84)
• Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.84)
• MoLoRA: Composable Specialization via Per-Token Adapter Routing (ICML 2026 · similarity 0.83)
• Learning to Solve Compositional Geometry Routing Problems (NeurIPS 2026 · similarity 0.83)
• VL-RouterBench: A Benchmark for Vision-Language Model Routing (CVPR 2026 · similarity 0.83)