Research map / Learning theory and algorithms / Prediction and decision losses
DISCO: Diversifying Sample Condensation for Accelerating Model Evaluation
Alexander Rubinstein, Benjamin Raible, Martin Gubri, Seong Joon Oh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add model signatures with Random Forest on uniform random samples (ablation) | direct evaluation on random samples · MMLU | MAE and rank: 3.45 -> 1.81 MAE; .916 -> .933 rank | Improved |
| select anchor points with high PDS instead of uniform random (ablation) | Sig. + RF · MMLU | MAE and rank: 1.81 -> 1.07 MAE; .933 -> .987 rank | Improved |
| replace chronological split by uniform split (ablation) | DISCO · MMLU | rank: .987 vs .986 | No clear effect |
| add task stratification of anchors (ablation) | DISCO with PDS · MMLU | rank: .978 vs .987 | No clear effect |
| reduce source models to 100 from 382 (ablation) | DISCO · MMLU | rank: .987 -> .969 | Hurt |
| add PCA dimensionality reduction to 256 dims (ablation) | DISCO · MMLU | rank: .918 -> .987 | Improved |
| use Random Forest instead of other prediction models (ablation) | DISCO · MMLU | rank: .987 | Improved |
| use kNN instead of Random Forest at 10 samples (ablation) | DISCO · MMLU | MAE and rank | Improved |
| add PDS selection with Random Forest on vision (ablation) | Sig. + RF · ImageNet-1k val | MAE and rank: 0.86 -> 0.63 MAE; .944 -> .969 rank | Improved |
| add model signatures with RF on uniform samples (vision) (ablation) | direct evaluation · ImageNet-1k val | MAE and rank: 3.03 -> 0.86 MAE; 0.652 -> 0.944 rank | Improved |
| use High JSD instead of High PDS with kNN (ablation) | DISCO · HellaSwag | MAE: 1.32 -> 1.50 MAE | Hurt |
Closest papers by meaning
- Ameliorate Spurious Correlations in Dataset Condensation (ICML 2024 · similarity 0.87)
- Diversified Batch Selection for Training Acceleration (ICML 2024 · similarity 0.86)
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.85)
- Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data (ICLR 2025 · similarity 0.85)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.85)
- Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
