Research map / Label-efficient and robust learning / Recognition and long-tail
Generative Modeling Reinvents Supervised Learning: Label Repurposing with Predictive Consistency Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes predictive consistency learning (PCL), which feeds noise-perturbed labels as extra input hints and enforces prediction consistency across noise levels, then uses multistep denoising-style inference for complex-label prediction.
Problem
Direct supervised prediction struggles when labels are complex and high-dimensional (dense segmentation, N-body trajectories, next-token prediction), since mapping learned features to high-fidelity label representations is hard.
Approach
Treat noisy labels as an auxiliary conditioning input alongside data, train the model so predictions from different noise levels agree with each other and with the true label (a triadic loss), and at inference iteratively re-noise and refine the prediction like gradual denoising.
Key results
- On constrained N-body simulation, PCL lowers GCN prediction error on the (3,2,1) configuration from 3.479 to 2.795 (x10^-2) versus supervised learning.
- On ADE20K with ResNet50dilated+PPM, PCL raises Score from 60.24 to 65.45 (Pixel Acc 78.98 to 82.33, mIoU 41.49 to 48.56).
- Full fine-tuning LLaMA-2-7B with PCL improves MMLU from 46.22 (SL) to 47.10 and CRASS from 58.29 to 59.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCL instead of supervised learning | GCN · Constrained N-body (3, 2, 1) | prediction error (×10−2): 3.479±0.110 -> 2.795±0.061 | Helped |
| Use PCL instead of supervised learning | GAT · Constrained N-body (3, 2, 1) | prediction error (×10−2): 3.351±0.111 -> 2.802±0.216 | Helped |
| Use PCL instead of supervised learning | GGNN · Constrained N-body (3, 2, 1) | prediction error (×10−2): 3.293±0.023 -> 2.974±0.011 | Helped |
| Use PCL instead of supervised learning | MobileNetV2dilated segmentation model · ADE20K | mIoU: 33.13 -> 35.27 | Helped |
| Use PCL instead of supervised learning | ResNet50dilated+PPM segmentation model · ADE20K | mIoU: 41.49 -> 48.56 | Helped |
| Use PCL instead of supervised learning | LLaMA2-7B full fine-tuning · MMLU | score: 46.22 -> 47.10 | Helped |
| Use PCL instead of supervised learning | LLaMA2-7B full fine-tuning · CRASS | score: 58.29 -> 59.48 | Helped |
| Use PCL instead of supervised learning | LLaMA2-7B full fine-tuning · BBH | score: 33.38 -> 34.75 | Helped |
| Remove the λ1 loss term while retaining the λ2 term (ablation) | PCL with GCN · Constrained N-body (3, 2, 1) | prediction error (×10−2): 2.795±0.061 -> 6.784±0.862 | Hurt |
| Remove the λ2 loss term (ablation) | PCL with GCN · Constrained N-body (3, 2, 1) | prediction error (×10−2): 2.881±0.110 -> 2.795±0.061 | Hurt |
| Remove the λ1 loss term while retaining the λ2 term (ablation) | PCL with GAT · Constrained N-body (3, 2, 1) | prediction error (×10−2): 2.802±0.216 -> 4.234±0.170 | Hurt |
| Remove the λ2 loss term (ablation) | PCL with GAT · Constrained N-body (3, 2, 1) | prediction error (×10−2): 3.001±0.074 -> 2.802±0.216 | Hurt |
Surprising findings
- Lambda2 self-consistency term alone (without the label-supervised term) performs poorly, e.g., GCN error jumps to 6.784 on the (1,2,0) config.
- Multistep inference error first decreases then rises for a period before decreasing again, requiring validation-based early stopping of the inference step count.
Limitations
- Training requires two forward passes per sample, roughly doubling training cost.
- Multistep inference accumulates errors, so the number of inference steps must be tuned on validation data.
- Evaluated on a limited set of tasks; gains on LLM fine-tuning are modest and label-hint granularity for text is token-level only.
Experimental setup
Datasets: ADE20K Alpaca-cleaned constrained N-body simulation data (Huang et al., 2022) unconstrained N-body simulation (Satorras et al., 2021)
Benchmarks: ADE20K constrained N-body simulation unconstrained N-body simulation MMLU BBH CRASS
Metrics: MSE pixel accuracy mIoU prediction error MAE
Baselines: traditional supervised learning label smoothing SL bagging ensemble GCN GAT GGNN LLaMA2-7B full fine-tuning (SL)
Builds on: consistency models denoising diffusion probabilistic models discrete diffusion models PPM (pyramid pooling module)
Closest papers by meaning
- Provably Consistent Partial-Label Learning (NeurIPS 2020 · similarity 0.89)
- Controller-Guided Partial Label Consistency Regularization with Unlabeled Data (AAAI 2024 · similarity 0.88)
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.87)
- Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning (AAAI 2024 · similarity 0.87)
- Generative Calibration of Inaccurate Annotation for Label Distribution Learning (AAAI 2024 · similarity 0.86)
- Pre-Trained Vision-Language Models as Noisy Partial Annotators (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.