Research map / Large language models / Language, safety and interpretability

RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RouterDC, a query-based LLM router trained with a sample-LLM and a sample-sample contrastive loss, which assembles seven 7-8B open LLMs and beats the best single model and prior routers on in-distribution and out-of-distribution tasks.

Problem

Existing LLM routers, trained with KL alignment to normalized scores, perform poorly when multiple LLMs are good for the same query and training is unstable for similar queries.

Approach

Train a small encoder with learnable LLM embeddings using a contrastive loss that pulls queries toward top-K performing LLMs and pushes them from bottom performers, plus a clustering-based contrastive loss that makes similar queries map to similar embeddings.

Key results

• In-distribution average accuracy 58.54% vs 54.56% for best single LLM (dolphin-2.9-llama3-8b) and 55.77% for LoraRetriever (clustering).
• Out-of-distribution average accuracy 45.85% vs 43.95% for best single LLM and 43.50% for LoraRetriever.
• Inference about 6x faster than Voting (7.97 vs 46.59 minutes on in-distribution tasks) since only one LLM is called per query.

Measured effects reported in the paper

Surprising findings

• Removing dolphin-2.9-llama3-8b drops HumanEval accuracy from 51.02 to 46.53, while removing other single LLMs has little effect, showing a single strong model dominates some tasks.
• Using only the sample-LLM loss yields unstable training; similar queries get dissimilar embeddings and are routed differently.

Limitations

• Only candidate LLMs of 7-8B parameters were evaluated due to limited compute; larger models like LLaMA-2-70B were not tested.
• Only query-based routing is studied; extension to chat contexts is left to future work.
• Reported improvements are averages over a small set of tasks and seven candidate LLMs.

Experimental setup

Datasets: MMLU GSM8K CMMLU ARC-C HumanEval PreAlgebra MBPP C-EVAL JavaScript RouterBench

Benchmarks: MMLU GSM8K CMMLU ARC-C HumanEval PreAlgebra MBPP C-EVAL

Metrics: accuracy average accuracy inference time

Baselines: Voting ZOOTER LoraRetriever CosineClassifier Mistral-7B dolphin-2.9-llama3-8b

Builds on: ZOOTER LoraRetriever mDeBERTaV3-base t-SNE k-means

Closest papers by meaning

• RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.89)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Cost-Aware Contrastive Routing for LLMs (NeurIPS 2025 · similarity 0.88)
• Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.88)
• BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.87)