Research map / Large language models / Language, safety and interpretability
Post Hoc Regression Refinement via Pairwise Rankings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RankRefine, a post hoc method that fuses a base regressor's prediction with a rank-based estimate from pairwise comparisons (from oracles, ChatGPT-4o, or humans) via inverse-variance weighting, reducing MAE in low-data regression without retraining.
Problem
Regression accuracy deteriorates in data-scarce regimes, and expert or LLM pairwise relative judgments are underused to improve continuous property predictions.
Approach
Converts pairwise comparisons between a query and labeled references into a Bradley-Terry maximum-likelihood rank-based estimate with Fisher-information variance, then fuses it with the regressor output by inverse-variance weighting, with a theoretical guarantee that MAE decreases for any finite-variance informative ranker.
Key results
- With 20 pairwise comparisons from ChatGPT-4o (ranking accuracy ~0.60-0.69), RankRefine reduced MAE on six TDC datasets, e.g., beta 0.895 on VDss and 0.934 on Solubility.
- Human-as-ranker study on UTKFace raised ranking accuracy to 0.759 and reduced MAE by about 5% (beta 0.954).
- Oracle experiments show improvement even at ranker accuracy around 0.55 with k=10 comparisons; RankRefine beats RbR across datasets and beats projection when ranker accuracy is between 0.5 and 0.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse regression predictions with pairwise rankings using inverse-variance weighting | Random forest regression with ChatGPT-4o rankings · Six TDC ADME datasets | β (post-refinement MAE / regression MAE): 0.957 ± 0.012; 0.934 ± 0.048; 0.895 ± 0.053; 0.970 ± 0.027; 0.971 ± 0.005; 0.937 ± 0.012 | Helped |
| Refine direct human age estimates using human pairwise judgments | Human age estimates · UTKFace age estimation | β (post-refinement MAE / regression MAE): 6.343 ± 0.610; 0.954 ± 0.046 | Helped |
| Inject constant bias into the regressor, increasing from B = 0% SD to B = 70% SD (ablation) | RankRefine on synthetic regression · Synthetic biased-regressor experiment | β: 0.8694 -> 1.0022 | Hurt |
| Restrict references to clustered labels rather than the full ground-truth range (ablation) | RankRefine on synthetic regression · Synthetic biased-sampling experiment, ranker accuracy 60% | β: 0.8343 -> 0.8842 | Hurt |
| Use disjoint reference-label and query-label ranges (ablation) | RankRefine on synthetic regression · Synthetic distribution-shift experiment, ranker accuracy 60% | β: 1.0170 | Hurt |
| Use partially overlapping reference-label and query-label ranges (ablation) | RankRefine on synthetic regression · Synthetic distribution-shift experiment, ranker accuracy 60% | β: 0.8700 | Helped |
| Increase the number of pairwise references (ablation) | RankRefine with oracle rankers · Molecular-property prediction | β | Helped |
| Use 30 rather than 20 pairwise references (ablation) | RankRefine with oracle rankers · Molecular-property prediction | β | No clear effect |
| Use a near-perfect ranker (accuracy >0.95) (ablation) | RankRefine with oracle rankers · Molecular-property prediction | β: >0.95 | Hurt |
| Use RankRefine instead of regression by re-ranking | Post-hoc regression refinement · All evaluated datasets | β | Helped |
| Use RankRefine instead of projection-based refinement | Post-hoc regression refinement · Ranker accuracy 0.50–0.95 | β: 0.50–0.95 | Helped |
| Use RankRefine instead of projection-based refinement with a nearly perfect ranker | Post-hoc regression refinement · Near-perfect ranker accuracy | β | Hurt |
Surprising findings
- Projection-based refinement outperforms RankRefine when rankers are nearly perfect (above about 0.95 accuracy).
- A ChatGPT-4o ranking accuracy of 60% on a private unpublished dataset suggests the LLM is not merely memorizing.
Limitations
- Theory assumes unbiased Gaussian independent errors, which may not hold for heavy-tailed or skewed noise.
- Performance depends on well-calibrated regressor uncertainty and accurate ranker variance estimates.
- Oracle experiments assume random ranking errors; real rankers may have systematic biases such as failing on extreme values.
Experimental setup
Datasets: TDC ADME benchmark (Caco-2, Clearance Microsome, Clearance Hepatocyte, Half Life, FreeSolv, Lipophilicity, PPBR, Solubil Smart Farming crop yield Student Performance International Education Cost UTKFace
Benchmarks: TDC ADME benchmark Smart Farming Student Performance International Education Cost UTKFace age estimation
Metrics: MAE normalized error beta pairwise ranking accuracy
Baselines: constrained optimization projection refinement regression by re-ranking (RbR)
Builds on: Bradley-Terry model inverse-variance weighting regression by re-ranking
Closest papers by meaning
- BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models (NeurIPS 2025 · similarity 0.84)
- CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction (NeurIPS 2025 · similarity 0.84)
- From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction (ICLR 2024 · similarity 0.84)
- Aligning Transformers with Continuous Feedback via Energy Rank Alignment (NeurIPS 2025 · similarity 0.83)
- From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery (AAAI 2024 · similarity 0.83)
- MIPT: Multilevel Informed Prompt Tuning for Robust Molecular Property Prediction (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.