Research map / Graph learning / Graph algorithms and clustering
Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RG-PT, a multi-objective hyperparameter selection method that builds a reliability DAG via Bradley-Terry scores and non-negative Lasso and applies DAGGER FDR-controlling testing, outperforming LTT and PT in prompt engineering and translation.
Problem
Hyperparameter selection (e.g., LLM prompt templates) that balances reliability constraints with cost objectives while retaining formal FDR guarantees and exploiting known structural reliability relations among candidates.
Approach
Model hyperparameter reliability as a DAG (reliability graph) learned from held-out data and prior pairwise preferences (Bradley-Terry scores for depth, non-negative Lasso for edges), then run DAGGER graph-structured FDR-controlling testing so that multiple configurations at the same depth are tested in parallel, enlarging the reliable set beyond the linear ordering used by Pareto testing.
Key results
- On prompt engineering, RG-PT achieved an average FDR of 0.089, 0.095, and 0.092 across sentiment, similarity, and word-in-context tasks at target delta=0.1, while returning shorter prompts than LTT and PT.
- On Fashion MNIST SVM tuning, RG-PT achieved a test recall of 0.332 versus 0.727 for LTT and PT, with classification error 0.286 versus 0.267.
- Across six experiment types, average test FDR stayed near or below the 0.1 target (e.g., 0.084 to 0.095).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RG-PT instead of LTT or PT | prompt selection · Sentiment analysis, sentence similarity, and word in context | shortest reliable prompt length | Helped |
| use RG-PT instead of LTT or PT | translation hyperparameter selection · WMT16 Romanian-English | test ROUGE-L | Helped |
| use RG-PT instead of LTT or PT | image detection hyperparameter selection · MS-COCO | object recall | Helped |
| use RG-PT instead of LTT or PT | SVM hyperparameter selection · Fashion MNIST | test recall: 0.727 -> 0.332 | Helped |
| use RG-PT instead of LTT or PT | SVM hyperparameter selection · Fashion MNIST | test classification error: 0.267 -> 0.286 | Hurt |
| vary RG depth D from 1 to 100 (ablation) | RG-PT · Prompt Engineering | average shortest reliable prompt length | Mixed |
| vary RG depth D from 1 to 100 (ablation) | RG-PT · Prompt Engineering | test FDR | No clear effect |
| increase the fraction of flipped pairwise priors f (ablation) | RG-PT · Prompt Engineering | shortest reliable prompt length | Hurt |
| increase the fraction of flipped pairwise priors f (ablation) | RG-PT · Prompt Engineering | test FDR | No clear effect |
| use highly corrupted priors with f > 0.6 (ablation) | RG-PT · Prompt Engineering | number of selected prompts | Hurt |
Surprising findings
- Misspecified priors that flip pairwise preferences for f > 0.6 cause RG-PT to return an empty set, while FDR remains controlled.
Limitations
- Applicable only to discrete hyperparameter spaces
- No theoretical results on power or sample efficiency
- Depth D and prior quality must be chosen; benefits depend on prior information
Experimental setup
Datasets: Instruction Induction Stanford Sentiment Treebank Semantic Textual Similarity Benchmark Word-in-Context WMT16 Romanian-English MS-COCO Fashion MNIST
Benchmarks: Instruction Induction WMT16 Romanian-English MS-COCO Fashion MNIST
Metrics: average prompt length of shortest reliable prompt FDR ROUGE-L recall classification error
Baselines: LTT PT
Builds on: Learn-then-Test Pareto Testing DAGGER Benjamini-Hochberg fixed sequence testing
Closest papers by meaning
- The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph (ICML 2025 · similarity 0.83)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.83)
- GraphRouter: A Graph-based Router for LLM Selections (ICLR 2025 · similarity 0.83)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.83)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.82)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.