Research map / Architectures and efficiency / Pruning and efficient tuning
Data Shunt: Collaboration of Small and Large Models for Lower Costs and Better Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Data Shunt, a confidence-based router that sends hard inputs from small models to large models, plus Prompt Pruning and 2-Stage Confidence Distillation to improve both cost and accuracy across text, image classification, and captioning.
Problem
Large models are costly to run or query via API, while small models sometimes match or beat large models on certain data distributions; the paper targets reducing large-model calls while maintaining or improving performance.
Approach
Route each input by the confidence of a small model: confident inputs stay with the small model, low-confidence inputs go to the large model with the small model's confidence injected into the prompt (Prompt Pruning), and the small model is improved by distilling from the large model using a two-stage confidence-gated scheme (2CD) to avoid catastrophic forgetting.
Key results
- On Amazon sentiment analysis, DS with fine-tuned BERT reaches 95.64% accuracy vs. 94.43% for ChatGPT while sending only 31.18% of queries to ChatGPT.
- On CIFAR-100-LT, DS reaches 63.25% overall accuracy vs. 58.18% for CLIP (large) with 66.10% of queries routed to the large model.
- On MS COCO captioning, DS improves mean BLEU to 54.52 vs. 54.10 for BLIP-2 while routing 65.36% of data to the large model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Data Shunt with fine-tuned BERT to route low-confidence inputs to ChatGPT | Fine-tuned BERT and ChatGPT · Amazon Product sentiment analysis | accuracy: 91.79% -> 95.64% | Helped |
| Use Data Shunt with fine-tuned BERT to route low-confidence inputs to ChatGPT | Fine-tuned BERT and ChatGPT · Amazon Product sentiment analysis | query proportion: 100% -> 31.18% | Helped |
| Route low-confidence inputs to CLIP using Data Shunt | ResNet-32 and CLIP · CIFAR-100-LT | overall accuracy: 58.18% -> 63.25% | Helped |
| Use confidence-based Data Shunt routing | ResNet-101, LSTM, and BLIP-2 (1.1B) · Microsoft COCO | mean BLEU: 54.10 -> 54.52 | Helped |
| Remove 2-Stage Confidence Distillation (ablation) | Data Shunt on CIFAR-100-LT · CIFAR-100-LT | overall accuracy: 63.25% -> 62.11% | Hurt |
| Add 2-Stage Confidence Distillation (ablation) | Data Shunt on CIFAR-100-LT · CIFAR-100-LT | query proportion: 66.10% -> 67.48% | Helped |
| Add Prompt Pruning to Data Shunt with 2-Stage Confidence Distillation (ablation) | Data Shunt on CIFAR-100-LT · CIFAR-100-LT | overall accuracy: 62.11% -> 61.63% | Hurt |
| Add Prompt Pruning to Data Shunt with 2-Stage Confidence Distillation (ablation) | Data Shunt on CIFAR-100-LT · CIFAR-100-LT tail | accuracy: 56.17% -> 53.31% | Hurt |
Surprising findings
- On CIFAR-100-LT, the small ResNet-32 outperforms the large CLIP on head-class data (70.25% vs. 60.00%).
- Small models with weaker standalone accuracy (TextCNN, LSTM) need to route 80-85% of queries to ChatGPT to match its accuracy, so the cost savings depend strongly on the small model's quality.
Limitations
- Large-model results on vision and captioning use CLIP and BLIP-2 (1.1B) rather than a stronger frontier model, and cost is measured as query proportion rather than actual monetary cost.
- Threshold selection requires validation tuning, and gains are small for some configurations (e.g., BLEU-3/4 fall below the large model).
- Experiments use a single small/large pair per modality and mainly classification tasks; generality beyond these is untested.
Experimental setup
Datasets: Amazon Product Data CIFAR-100-LT Microsoft COCO
Benchmarks: Amazon Product Data sentiment analysis CIFAR-100-LT Microsoft COCO
Metrics: accuracy query proportion BLEU-1 BLEU-2 BLEU-3 BLEU-4
Baselines: ChatGPT TextCNN LSTM fine-tuned BERT ResNet-32 CLIP BLIP-2 Show and Tell-style LSTM captioner
Builds on: knowledge distillation LLM cascade
Closest papers by meaning
- Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models (ICLR 2025 · similarity 0.87)
- DataDecide: How to Predict Best Pretraining Data with Small Experiments (ICML 2025 · similarity 0.87)
- Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.86)
- DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration (NeurIPS 2025 · similarity 0.86)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.86)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.