Research map / Learning theory and algorithms / Kernels and regression theory

Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

ICML 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Statistical perspectives on generalization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes model steering (learning with a reference model) via DRRho risk minimization, gives generalization bounds, and applies it to CLIP training as DRRho-CLIP, which outperforms heuristic RHO-loss sampling and has a better scaling law than OpenCLIP.

Problem

Learning with a reference model (RHO-loss-based data selection/weighting) is used heuristically in practice, but its generalization benefits and optimal selection/weighting rules are not theoretically understood.

Approach

Apply distributionally robust optimization to the RHO loss (target loss minus reference loss) across the whole training set, yielding a DRRho risk whose generalization bound has a variance term that the reference model can reduce; instantiate for CLIP via KL-regularized DRO contrastive loss optimized with SogCLR.

Key results

• On DFN-192M with OpenAI ViT-B/32 as reference, DRRho-CLIP ViT-B/16 reaches 68.84% ImageNet top-1 vs 67.37% FastCLIP and 63.32% reference model.
• DRRho-CLIP with 50% of training data is reported to match FastCLIP on full data when using ViT-B/32 (WIT-400M) reference on DFN-12M setting.
• DRRho-CLIP shows a smaller scaling-law exponent beta than OpenCLIP, indicating a better scaling trend.

Measured effects reported in the paper

Surprising findings

• DRRho-CLIP surpasses the reference model's accuracy even though the reference is weaker, i.e., weak-to-strong generalization.
• Heuristic JEST and JEST (Top-k) underperform the FastCLIP baseline on DFN-192M despite using reference models.

Limitations

• Theory relies on VC-dimension based bounds and a variance argument that is assumed rather than fully verified; reference model assumed sufficiently trained.
• Scaling law fit uses lowest error across dataset sizes as estimate, with limited model sizes (ViT-B/32, B/16, L/14).

Experimental setup

Datasets: CC12M DFN-192M DFN-12M DFN-2B

Benchmarks: ImageNet-1K zero-shot Datacomp (38 tasks)

Metrics: zero-shot top-1 accuracy Datacomp average scaling law exponent beta

Baselines: OpenCLIP FastCLIP JEST JEST (Top-k) MobileCLIP FastCLIP (w/ Distillation)

Builds on: RHO loss SogCLR FastCLIP JEST MobileCLIP

Closest papers by meaning

• Curvature Tuning: Provable Training-free Model Steering From a Single Parameter (NeurIPS 2025 · similarity 0.85)
• Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models (NeurIPS 2024 · similarity 0.85)
• Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.85)
• Gradient-Guided Annealing for Domain Generalization (CVPR 2025 · similarity 0.85)
• Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.85)
• Understanding Layer Patching in Model Size Interpolation (NeurIPS 2026 · similarity 0.84)