Research map / Probabilistic and scientific modeling / Variational and generative sampling
Structural Entropy Guided Probabilistic Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEPC, a probabilistic coding model that adds a structural-entropy regularization loss over latent embeddings and a probabilistic encoding tree that extends it to regression via soft labels.
Problem
Existing IB-based probabilistic embedding regularizers constrain each latent variable individually (or against labels) and ignore structural relations between latent variables; structural entropy has also been limited to classification.
Approach
Maximize structural entropy of a graph built from latent embeddings (grouped by class in a three-tier encoding tree) as a regularizer, and make it applicable to regression by softening binned labels into soft assignments with a probabilistic encoding tree.
Key results
- SEPC with RoBERTa averages 63.99% over 10 classification datasets vs 62.47% for SPC, and 71.80 average on STS-B/Claire regression vs 70.84 for SPC.
- Removing the structural entropy loss lowers average classification score by 2.41 (BERT) and 1.99 (RoBERTa) points.
- Under 30% label noise SEPC keeps an average of 63.32 vs 61.23 for SPC (BERT-based comparison in robustness table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add structural entropy regularization (ablation) | SEPC with BERT · Classification tasks (10 datasets) | Average macro metric (%): 59.82 -> 62.23 | Helped |
| Add structural entropy regularization (ablation) | SEPC with RoBERTa · Classification tasks (10 datasets) | Average macro metric (%): 62.00 -> 63.99 | Helped |
| Use soft labels and a probabilistic encoding tree instead of nearest-bin assignments and a normal encoding tree (ablation) | SEPC with RoBERTa · Regression tasks (STS-B and Claire) | Average (%): 71.17 -> 71.80 | Helped |
| Add structural entropy regularization (ablation) | SEPC with BERT · Emoji | F1 (%): 22.49 -> 24.85 | Helped |
| Add structural entropy regularization (ablation) | SEPC with RoBERTa · Emotion | F1 (%): 79.25 -> 79.82 | Helped |
| Increase the regularization weight to γ = 10 | SEPC · Classification tasks | Performance | Hurt |
| Train with label noise at 10%, 20%, or 30% | SEPC with RoBERTa · Classification tasks (10 datasets) | Average macro metric (%) | Mixed |
Limitations
- Regression is handled by discretizing labels into bins, with the number of bins r chosen manually (5 for STS-B, 4 for Claire)
- Hyperparameters beta and gamma are grid-searched over a small set
- Only text/NLU tasks evaluated
Experimental setup
Datasets: Emoji Emotion Hate Irony Offensive Sentiment Stance ISEAR MELD GoEmotions STS-B Claire
Benchmarks: Emoji Emotion Hate Irony Offensive Sentiment Stance ISEAR MELD GoEmotions STS-B Claire
Metrics: macro-F1 recall Pearson correlation Spearman correlation
Baselines: SVM FastText BiLSTM GPT-3.5 CE CE+CP CE+AT CE+SCL VIB MINE-IB MEIB SPC
Builds on: Information Bottleneck VIB SPC MEIB structural information theory
Closest papers by meaning
- Structured Probabilistic Coding (AAAI 2024 · similarity 0.91)
- Minimum Description Length and Generalization Guarantees for Representation Learning (NeurIPS 2023 · similarity 0.86)
- SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models (AAAI 2025 · similarity 0.86)
- Structural-Entropy-Based Sample Selection for Efficient and Effective Learning (ICLR 2025 · similarity 0.83)
- Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models (NeurIPS 2025 · similarity 0.83)
- Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.