Research map / Continual learning / Online and replay-based
Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PGM, a group-wise probabilistic mask selection method for task-incremental learning that samples candidate masks per parameter group with Gumbel-Softmax, outperforming WSN and other parameter-isolation and regularization baselines on CIFAR-100 and TinyImageNet.
Problem
Parameter-isolation incremental learning methods score parameters independently when selecting task-specific masks, ignoring parameter dependencies and over-relying on newly allocated parameters, which causes capacity waste.
Approach
Partition parameters into groups, each with multiple candidate binary masks and learned probabilities over mask sizes and patterns; use Gumbel-Softmax to make the two-stage categorical mask sampling differentiable, and initialize a new task's mask distribution from similarity to prior task masks.
Key results
- PGM reaches 77.29% ACC on Split CIFAR-100, 62.56% on CIFAR-100 Superclass and 70.99% on TinyImageNet, the highest among compared methods.
- On CIFAR-100 Superclass, PGM uses 50.38% capacity versus 80.93% for WSN, with zero forgetting (BWT 0.0).
- Ablation: the group module lowers CAP from 99.13% to 65.97% on Split CIFAR-100 while mask initialization further lowers it to 63.51%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PGM instead of SPG | Incremental learning · CIFAR-100 Split | ACC: 77.12(±0.42) -> 77.29(±0.31) | Helped |
| Use PGM instead of SPG | Incremental learning · CIFAR-100 Superclass | ACC: 62.04 (±0.39) -> 62.56(±0.71) | Helped |
| Use PGM instead of RP2F | Incremental learning · TinyImageNet | ACC: 70.32(±0.29) -> 70.99(±0.54) | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · CIFAR-100 Split | ACC: 76.38 -> 76.98 | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · CIFAR-100 Superclass | ACC: 61.79 -> 62.04 | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · TinyImageNet | ACC: 69.06 -> 70.13 | Helped |
| Add Mask Initialization to PGM with Group (ablation) | PGM · CIFAR-100 Split | ACC: 76.98 -> 77.29 | Helped |
| Add Mask Initialization to PGM with Group (ablation) | PGM · CIFAR-100 Superclass | ACC: 62.04 -> 62.56 | Helped |
| Add Mask Initialization to PGM with Group (ablation) | PGM · TinyImageNet | ACC: 70.13 -> 70.99 | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · CIFAR-100 Split | CAP: 99.13 -> 65.97 | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · CIFAR-100 Superclass | CAP: 80.93 -> 54.25 | Helped |
| Add Group to the method without Group or Mask Initialization (ablation) | PGM · TinyImageNet | CAP: 92.03 -> 82.39 | Helped |
Surprising findings
- Dependencies appear predominantly localized, with interactions concentrated among adjacent parameters, suggesting global dependencies may often be unnecessary.
Limitations
- Independent sampling per group ignores interdependencies between groups.
- Evaluated only in task-incremental setting with multi-head configuration on small CNN backbones.
Experimental setup
Datasets: Split CIFAR-100 CIFAR-100 Superclass Split TinyImageNet
Benchmarks: Split CIFAR-100 CIFAR-100 Superclass Split TinyImageNet
Metrics: ACC CAP BWT
Baselines: Multitask La-MAML GPM FS-DGPM RP2F PackNet SupSup WSN SPG
Builds on: WSN SupSup PackNet Gumbel-Softmax
Closest papers by meaning
- Parameter-Level Soft-Masking for Continual Learning (ICML 2023 · similarity 0.88)
- Parameter-Masked Decoupled Optimization for Cross-Domain Class-Incremental Learning (ICML 2026 · similarity 0.86)
- Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (NeurIPS 2020 · similarity 0.85)
- MetaFSCIL: A Meta-Learning Approach for Few-Shot Class Incremental Learning (CVPR 2022 · similarity 0.85)
- Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method (ICML 2024 · similarity 0.84)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.