Research map / Label-efficient and robust learning / Recognition and long-tail

Structured IB: Improving Information Bottleneck with Structured Feature Learning

AAAI 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Structured IB (SIB), which augments an IB Lagrangian main encoder with sequentially trained auxiliary encoders (kept decorrelated via a discriminator-based MI penalty) and a learned weighted sum, improving I(Z,Y) and accuracy with fewer parameters on MNIST and CIFAR-10.

Problem

IB Lagrangian methods rely on variational bounds or neural estimators whose approximations and implementation choices can cause information loss, limiting the informativeness of the learned representation.

Approach

Split the feature extractor into a main encoder trained with the IB Lagrangian plus auxiliary encoders that are trained with the IB Lagrangian to capture missed information while being penalized for mutual information with previous features; the features are combined by a weighted sum with the decoder frozen.

Key results

• On MNIST with K=2, SVIB-style structured models improve accuracy and I(Z,Y) over the single-encoder IB Lagrangian while using fewer parameters (reported e.g. NIB at 0.9498 weight-pattern accuracy context; estimator variants reach 0.9853 accuracy at K=3 vs 0.9720 at K=1 for MINE).
• For CIFAR-10 with MINE, accuracy rises from 0.7659 (K=1) to 0.7821 (K=3); KNIFE rises from 0.7710 to 0.7832.
• The main encoder consistently receives the largest aggregation weight, and weights for variational methods approximately sum to 1 without explicit normalization.

Measured effects reported in the paper

Surprising findings

• Encoder dropout experiments show I(X,Z) increases as auxiliary encoders are removed, suggesting auxiliary encoders also contribute to compression.
• For neural estimator methods (MINE, KNIFE), aggregation weights accumulate to roughly K rather than 1.

Limitations

• Linear weighted-sum aggregation may oversimplify feature interactions.
• Theoretical guarantees rely on Gaussian assumptions and only cover I(Z,Y), not I(X,Z); the compression behavior is shown empirically.
• Performance plateaus as the number of encoders K grows, and estimator-based methods suffer convergence issues (NaN values, near-random accuracy at some beta).

Experimental setup

Datasets: MNIST CIFAR-10

Benchmarks: MNIST CIFAR-10

Metrics: accuracy I(Z,Y) I(X,Z) number of parameters

Baselines: VIB sqVIB NIB MINE KNIFE

Builds on: Variational Information Bottleneck Nonlinear Information Bottleneck Square IB MINE KNIFE

Closest papers by meaning

• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.89)
• Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.86)
• Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning (ICML 2024 · similarity 0.85)
• A Rate-Distortion View of Uncertainty Quantification (ICML 2024 · similarity 0.85)
• Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.85)
• Differentiable Information Bottleneck for Deterministic Multi-view Clustering (CVPR 2024 · similarity 0.84)