Research map / Learning theory and algorithms / Prediction and decision losses
Learning to Reject with a Fixed Predictor: Application to Decontextualization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning a rejector for a fixed predictor using a convex exponential surrogate of the induced rejection loss with an H-consistency bound, applied to decontextualization with a 2,000-example manually labeled dataset.
Problem
Classification with a reject option for a fixed sequence-to-sequence predictor in NLP tasks where many outputs are correct, so the full set of correct labels is unavailable for training the rejector.
Approach
Fix the predictor and learn only a rejector r(x) from binary correctness annotations of predictor outputs, minimizing an exponential surrogate loss of the induced rejection loss; with the parameter constraint 2βc/α = Ī_c the surrogate has a built-in threshold at 0 and an H-consistency bound of square-root form.
Key results
- At cost c=0.07 the surrogate loss halves the error rate (11.00% to 5.1%) while keeping 90.6% coverage, versus 64.4% coverage for the cross-entropy baseline at ~5.1% error.
- The surrogate loss stays close to the theoretical coverage limit (93.6% at 5% error) for the lower precision levels.
- Standard deviations across cross-validation splits are substantially smaller for the surrogate loss than for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the fixed-predictor rejector with the surrogate rejection loss instead of cross-entropy | T5X 1.1 XXL decontextualization rejector · Decontextualization, near 5.1% error | coverage: 90.6% coverage vs. 64.4% coverage | Helped |
| Use the surrogate rejection loss rather than confidence-based MaxProb thresholding | MT5 XXL decontextualization predictor · Decontextualization at lower-half precision levels | coverage: 10-25% improvement | Helped |
Limitations
- Requires manually labeled correctness annotations for each fixed predictor, which is costly and not reusable across predictors.
- Theoretical guarantee holds only under the constraint 2βc/α = Ī_c, leaving one parameter to be set arbitrarily or by search.
- Evaluation limited to a single task (decontextualization) and a single predictor.
Experimental setup
Datasets: English Wikipedia decontextualization data manually labeled decontextualization dataset (2,000 examples)
Benchmarks: decontextualization rejection task (2,000 annotated examples)
Metrics: coverage at target precision precision
Baselines: MaxProb cross-entropy rejector
Builds on: Cortes et al. learning with rejection surrogate loss H-consistency bounds MaxProb selective classification
Closest papers by meaning
- Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer (ICML 2025 · similarity 0.84)
- Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.84)
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · similarity 0.84)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.84)
- Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders (NeurIPS 2025 · similarity 0.83)
- Implicit Optimization Bias of Next-token Prediction in Linear Models (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.