Research map / Large language models / Reasoning and chain of thought
Cost-aware LLM-based Online Dataset Annotation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaMVo, an online, cost-aware weighted majority voting framework that adaptively selects a subset of LLMs per instance using LinUCB-based confidence lower bounds and a Bayesian Beta-mixture estimator, evaluated on MMLU and IMDB.
Problem
Majority voting over multiple LLMs improves labeling reliability for automated dataset annotation but incurs high querying costs, and existing cost-aware routing methods require pre-trained routers or ground-truth labels.
Approach
Adaptively select the cheapest LLM subset per instance whose weighted majority-vote confidence (computed from contextual LinUCB lower confidence bounds, Bayesian Beta-mixture posterior and Laplace-smoothed accuracy) exceeds a user threshold delta, updated online without ground truth or pre-training.
Key results
- On MMLU, CaMVo at delta=0.97 and k_min=1 reaches 88.35% accuracy at $6.67 per million tokens versus 88.18% and $9.14 for full majority vote.
- On IMDB, CaMVo at delta=0.997 and k_min=1 reaches 95.45% accuracy at $2.83 cost versus 95.61% and $6.29 for the baseline (less than half the cost).
- On IMDB, the baseline ensemble underperforms the best single model (95.61% vs 95.68%), suggesting correlated LLM errors limit majority voting gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online cost-aware subset voting instead of querying all LLMs | CaMVo on MMLU test at δ=0.97, k_min=1 · MMLU test | accuracy: 88.18 -> 88.35 | Helped |
| Use online cost-aware subset voting instead of querying all LLMs | CaMVo on MMLU test at δ=0.97, k_min=1 · MMLU test | cost: 9.14 -> 6.67 | Helped |
| Use online cost-aware subset voting instead of querying all LLMs | CaMVo on IMDB at δ=0.997, k_min=1 · IMDB Movie Reviews | accuracy: 95.62 -> 95.45 | Hurt |
| Use online cost-aware subset voting instead of querying all LLMs | CaMVo on IMDB at δ=0.997, k_min=1 · IMDB Movie Reviews | cost: 6.29 -> 2.83 | Helped |
| Raise the confidence threshold δ | CaMVo on MMLU test, k_min=1 · MMLU test | cost: 0.70 -> 9.14 | Mixed |
| Require at least three queried LLMs instead of allowing one | CaMVo on MMLU test at δ=0.75 · MMLU test | accuracy: 68.80 -> 81.24 | Helped |
| Require at least three queried LLMs instead of allowing one | CaMVo on IMDB at δ=0.90 · IMDB Movie Reviews | accuracy: 94.24 -> 95.06 | Helped |
| Require at least three queried LLMs instead of allowing one | CaMVo on IMDB at δ=0.90 · IMDB Movie Reviews | cost: 0.10 -> 0.81 | Hurt |
Surprising findings
- On IMDB, full majority voting underperforms the best individual model (GPT-4o), attributed to correlated outputs violating the independence assumption.
- On MMLU, CaMVo sometimes surpasses the ground-truth-based Pareto frontier of weighted subset voting in cost-accuracy space.
Limitations
- Relies on conditional independence of LLM outputs; performance degrades when LLM errors are correlated, as on IMDB at higher cost regimes.
- Majority voting ground-truth-free estimates use predicted labels rather than true labels, so accuracy estimates are proxies.
- Evaluated on only two datasets with a fixed set of LLMs and a single random permutation in main experiments.
Experimental setup
Datasets: MMLU IMDB Movie Reviews
Benchmarks: MMLU IMDB Movie Reviews
Metrics: accuracy cost per million input tokens precision recall F1
Baselines: Majority Vote (all LLMs) Baseline Method (online weighted majority) Individual LLMs
Builds on: LinUCB weighted majority voting GLAD
Closest papers by meaning
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.87)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.87)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.86)
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking (NeurIPS 2025 · similarity 0.86)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.85)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.