Research map / Label-efficient and robust learning / Recognition and long-tail
Structured Probabilistic Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes structured probabilistic coding (SPC), an encoder-only variational probabilistic representation learner with a batch-level class-entropy regularizer, to improve fine-tuned BERT/RoBERTa on 10 classification and 2 regression NLU tasks.
Problem
Encoder-decoder information-bottleneck probabilistic embeddings can lose task-relevant information through injected latent noise, and they lack structure from the target label space, which hurts generalization with limited, biased, or noisy labels.
Approach
An encoder-only probabilistic coding module that outputs a Gaussian in the label space and performs prediction with a non-parametric softmax/MSE, combined with a structured regularizer that maximizes the batch-estimated entropy of the class-probability marginals to encourage uniform class coverage.
Key results
- With RoBERTa, SPC averages 62.35 on 10 classification tasks vs 58.43 for CE (+3.9 points), and the best among compared methods.
- On 2 regression tasks with RoBERTa, SPC averages 70.84 vs 69.15 for MSE (+1.7 points).
- Under 10/20/30% label noise, SPC improves average classification performance over CE by 1.9, 2.2 and 1.6 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPC instead of CE | RoBERTa · Classification tasks (10-task average) | average task score (%): +3.9% | Helped |
| Use SPC instead of MSE | RoBERTa · Regression tasks | average performance (%): +1.7% | Helped |
| Add structured regularization to probabilistic coding (ablation) | SPC with RoBERTa · Classification tasks (10-task average) | average task score (%): 62.35 -> 61.20 | Helped |
| Add probabilistic coding to the model with structured regularization (ablation) | SPC with RoBERTa · Classification tasks (10-task average) | average task score (%): 62.35 -> 58.43 | Helped |
| Add probabilistic coding instead of using MSE (ablation) | RoBERTa · Regression tasks | average performance (%): 1.7% | Helped |
| Use SPC instead of CE | RoBERTa · HatEval | macro-F1: +14.3% | Helped |
| Use SPC instead of CE | RoBERTa · IronyEval | ironic-class F1: +7.3% | Helped |
| Use SPC instead of CE | RoBERTa · Noisy-label classification tasks, 10% noise | average task score (%): +1.9% | Helped |
| Use SPC instead of CE | RoBERTa · Noisy-label classification tasks, 20% noise | average task score (%): +2.2% | Helped |
| Use SPC instead of CE | RoBERTa · Noisy-label classification tasks, 30% noise | average task score (%): +1.6% | Helped |
| Use SPC instead of other objectives | RoBERTa output representations · Most evaluated datasets | silhouette coefficient or adjusted rand index | Helped |
| Use SPC instead of comparison methods | RoBERTa · EmotionEval→GoEmotions, ISEAR→GoEmotions, and MELD→GoEmotions | task score (%) | Helped |
Surprising findings
- Zero-shot GPT-3.5 performs far below fine-tuned encoders on most tasks (47.21 average vs 58.43 for CE with RoBERTa).
Limitations
- Gains on some tasks are small or absent (e.g., EmotionEval and SentiEval are roughly on par with baselines), and IronyEval is better for MINE-IB/GPT-3.5.
- Hyperparameters beta and gamma are tuned per dataset from a grid, with significant per-task variation.
- Evaluated only on English tweet and emotion datasets with BERT-base and RoBERTa-base backbones.
Experimental setup
Datasets: EmojiEval EmotionEval HatEval IronyEval OffensEval SentiEval StanceEval ISEAR MELD GoEmotions STS-B CLAIRE
Benchmarks: EmojiEval EmotionEval HatEval IronyEval OffensEval SentiEval StanceEval ISEAR MELD GoEmotions STS-B CLAIRE
Metrics: macro-F1 F1 of ironic class macro recall Spearman correlation Pearson correlation silhouette coefficient adjusted rand index accuracy under label noise
Baselines: CE/MSE CE+CP CE/MSE+AT CE+SCL VIB MINE-IB MEIB SVM FastText BiLSTM GPT-3.5
Builds on: variational information bottleneck (VIB) MEIB supervised contrastive learning confidence penalty
Closest papers by meaning
- Structural Entropy Guided Probabilistic Coding (AAAI 2025 · similarity 0.91)
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.84)
- Zero-Shot Performance Prediction for Probabilistic Scaling Laws (NeurIPS 2025 · similarity 0.83)
- Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions (ICML 2024 · similarity 0.83)
- SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models (AAAI 2025 · similarity 0.83)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.