Research map / Large language models / AI and society
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDHF, a utility-guided two-stage cascade that learns from Copilot accept/reject telemetry to hide suggestions likely to be rejected, evaluated retrospectively on 535 programmers.
Problem
When should a code-recommendation system like GitHub Copilot display an inline suggestion, given that many suggestions are rejected, wasting latency and programmer verification time?
Approach
A utility-theoretic criterion shows a suggestion only when its acceptance probability exceeds a threshold P*; CDHF implements this with a cascade where a code-only stage-1 model decides to hide without generating the suggestion, and otherwise a stage-2 model using the suggestion decides, learned from implicit telemetry feedback rather than RLHF.
Key results
- CDHF can hide 25.3% of shown suggestions while guaranteeing 94.7% of them would have been rejected, and avoids generating 12.9% of suggestions.
- XGBoost acceptance model achieves 0.780 AUC and 81.1% accuracy across users; personalized model reaches 0.795 AUC.
- Using acceptance-probability ranking on HumanEval shows bias toward short suggestions, with the top-ranked suggestion often in the first few lines (over 60 of 164 problems for k=1 in [0,0.2] normalized length).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CDHF to hide suggestions below the acceptance-based display threshold | Copilot · CDHF retrospective test set | suggestions hidden (%): 25.3% hidden; 94.7% would have been rejected; 12.9% not generated | Helped |
| use CDHF to hide suggestions at the no-latency-concern operating point | Copilot · CDHF retrospective test set | suggestions hidden (%): 52.9% hidden; 91.3% would have been rejected | Helped |
| hide 52.9% of suggestions with CDHF | Copilot · CDHF counterfactual test set | acceptance rate (%): 22.5% -> 45.6% (+23.1 point increase) | Helped |
| include programmer ID and split across sessions for personalization | XGBoost acceptance predictor · Copilot acceptance prediction | AU-ROC: 0.780 -> 0.795 | Helped |
| remove suggestion features (ablation) | XGBoost acceptance predictor · Copilot acceptance prediction, session split | AU-ROC: 0.795 -> 0.631 | Hurt |
| include programmer latent state (ablation) | Programmer acceptance predictor · 21-participant study | accuracy: 61.9 ± 1.9 -> 83.6 ± 2.4 | Helped |
| rank candidate suggestions by predicted acceptance | Copilot suggestions · HumanEval with solution lines in the prompt (k>0) | length of highest-scored suggestion: over 60 of the 164 problems in [0,0.2]; at least 40 problems select the first token | Hurt |
| rank candidate suggestions by predicted acceptance with docstring-only prompts (k=0) | Copilot suggestions · HumanEval, k=0 | normalized length distribution of highest-rated suggestion: p-value 0.53 (KS=0.06) | No clear effect |
| increase XGBoost stage 2 training data from 1% to 25% | XGBoost stage 2 · Acceptance prediction sample complexity | AU-ROC: 0.69 -> 0.75 | Helped |
Surprising findings
- Optimizing suggestion ranking for acceptance favors short, partial suggestions, degrading quality.
- Including the programmer's latent state raises acceptance prediction accuracy from 61.9% to 83.6% in a 21-participant study.
Limitations
- Evaluation is retrospective only; no user study verifying productivity gains.
- Acceptance rate may be gamed (Goodhart's law) and rejected suggestions may still be useful.
- Telemetry lacks latent programmer state, so the deployed model cannot use it.
Experimental setup
Datasets: Copilot telemetry logs from 535 Microsoft programmers HumanEval user study telemetry of 21 participants
Benchmarks: HumanEval Copilot telemetry test split
Metrics: AUC accuracy macro F1 true negative rate fraction of suggestions hidden expected calibration error counterfactual acceptance rate
Baselines: Logistic Regression Random Forest FCNN QEBC
Builds on: GitHub Copilot RLHF learning to defer CodeBERTa
Closest papers by meaning
- Copilot Arena: A Platform for Code LLM Evaluation in the Wild (ICML 2025 · similarity 0.84)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.83)
- Towards Practical Defect-Focused Automated Code Review (ICML 2025 · similarity 0.83)
- Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation (NeurIPS 2024 · similarity 0.83)
- TOM-SWE: User Mental Modeling For Software Engineering Agents (ICML 2026 · similarity 0.82)
- Mecha-nudges for Machines (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.