Research map / Large language models / Instruction and fine-tuning
Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks LLMs for programming hint generation across quality, cost, time and privacy, and proposes in-browser hints (WebLLM, 4-bit Llama-3-8B and Phi-3-3.8B) fine-tuned with GPT-4 synthetic data.
Problem
Real-world deployment of LLM-based programming feedback is limited by running cost, latency and learner data privacy, not only feedback quality.
Approach
Run small fine-tuned LLMs directly in the learner's browser via in-browser inference (WebLLM), using a LoRA fine-tuning pipeline on GPT-4-generated synthetic buggy programs, repairs and hints to close much of the quality gap with GPT models while eliminating cost and data-sharing.
Key results
- Fine-tuned Llama-3-8B-dom-web reaches 64% HGood on all domains vs 21% for its base web version and 34% for GPT-3.5 (16-bit GPT-4 reaches 89%).
- Llama-3-8B-dom surpasses GPT-4o mini on all domains and comes close to GPT-4 on IntroPyNUS (96% HGood vs 100%).
- In-browser hint inference on a laptop-class GPU (RTX 3060) takes about 48 s for Llama-3-8B-dom-web, and as low as 20 s for Phi-3-3.8B-dom-web on M3 Max, with zero running cost per hint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3-8B on domain-specific synthetic data | Llama-3-8B · All domains | HGood (%): 36 -> 77 | Helped |
| Fine-tune Phi-3-3.8B on domain-specific synthetic data | Phi-3-3.8B · All domains | HGood (%): 27 -> 51 | Helped |
| Convert fine-tuned Llama-3-8B to 4-bit WebLLM (ablation) | Llama-3-8B-dom · All domains | HGood (%): 77 -> 64 | Hurt |
| Convert fine-tuned Phi-3-3.8B to 4-bit WebLLM (ablation) | Phi-3-3.8B-dom · All domains | HGood (%): 51 -> 36 | Hurt |
| Fine-tune Llama-3-8B on domain-specific synthetic data | Llama-3-8B · BasicAlgo | HGood (%): 32 -> 64 | Helped |
| Fine-tune Phi-3-3.8B on domain-specific synthetic data | Phi-3-3.8B · BasicAlgo | HGood (%): 36 -> 48 | Helped |
| Fine-tune Llama-3-8B on domain-specific synthetic data | Llama-3-8B · IntroPyNUS | HGood (%): 52 -> 96 | Helped |
| Fine-tune Phi-3-3.8B on domain-specific synthetic data | Phi-3-3.8B · IntroPyNUS | HGood (%): 32 -> 68 | Helped |
| Fine-tune Llama-3-8B on domain-specific synthetic data | Llama-3-8B · KarelAlgo | HGood (%): 24 -> 71 | Helped |
| Fine-tune Phi-3-3.8B on domain-specific synthetic data | Phi-3-3.8B · KarelAlgo | HGood (%): 14 -> 38 | Helped |
| Use n_r=3 instead of n_r=10 (ablation) | Llama-3-8B-dom-web · All domains | HGood (%): 64 (10) -> 59 (14) | Hurt |
| Use n_r=3 instead of n_r=10 (ablation) | Phi-3-3.8B-dom-web · All domains | HGood (%): 36 (14) -> 33 (14) | Hurt |
Surprising findings
- Web conversion (4-bit quantization) causes a much larger quality drop for Phi-3 models (aggregate HGood 51% to 8%) than for Llama-3 models (77% to 21% for base... see table; Llama-3 dom drops 77% to 64%).
Limitations
- Only Llama-3-8B and Phi-3-3.8B tested as base models; no comparison with smaller or programming-specialized models.
- Only WebLLM engine evaluated; ONNX Runtime not compared.
- Limited to Python programming feedback scenario and sustainability/energy not systematically evaluated.
Experimental setup
Datasets: BasicAlgo IntroPyNUS KarelAlgo
Benchmarks: BasicAlgo IntroPyNUS KarelAlgo
Metrics: HGood% RPass% REdit inference time cost per hint
Baselines: GPT-4 GPT-4o GPT-4o mini GPT-3.5 Llama-3-8B Phi-3-3.8B
Builds on: LoRA chain-of-thought WebLLM
Closest papers by meaning
- Teaching Language Models to Critique via Reinforcement Learning (ICML 2025 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.87)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.87)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.87)
- Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming (NeurIPS 2024 · similarity 0.87)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.