Research map / Large language models / Agents, code and math
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Lanbo Lin, Jiayao Liu, Tianyuan Yang, Li Cai, Yuanwu Xu, Lei Wei, Sicong Xie, Guannan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full two-layer JADE instead of vanilla pointwise LLM-as-a-judge | JADE · BizBench human alignment | Pearson r: 0.667 -> 0.858 | Improved |
| Add expert skill activation (Layer 1) (ablation) | JADE · BizBench human alignment | Pearson r: 0.769 -> 0.844 | Improved |
| Add report-specific checklist (Layer 2) (ablation) | JADE · BizBench human alignment | Pearson r: 0.769 -> 0.786 | Improved |
| Combine both layers rather than either alone (ablation) | JADE · BizBench human alignment | Pearson r: 0.844 -> 0.858 | Improved |
| Use a flat generated checklist instead of vanilla judge (ablation) | LLM judge · BizBench human alignment | Pearson r: 0.667 -> 0.769 | Improved |
| Swap JADE backbone from GPT-5-0807 to GPT-4.1 or Gemini-3-flash (ablation) | JADE · BizBench human alignment | Pearson r: 0.858 -> 0.827 (GPT-4.1); 0.858 -> 0.841 (Gemini-3-flash) | Hurt |
| Add evidence verification agent to JADE (ablation) | JADE · BizBench human alignment | Pearson r: 0.844 -> 0.858 | Improved |
| Add domain skills summarized from rubrics (ablation) | JADE · HealthBench subset (n=474) | Spearman rho: 0.154 -> 0.228 | Improved |
| Add domain skills on HealthBench after trimming 20% of items with specific clinical details (ablation) | JADE · HealthBench trimmed 20% | Spearman rho: 0.187 -> 0.530 | Improved |
| Apply JADE with auto-summarized skills to sports domain | JADE · DR.BENCH Sports & competitions | Pearson r: 0.478 (vs 0.736 overall) | Hurt |
| Enable tool use for agents | Claude Opus 4.5 · BizBench final score | final score (%): 32.1 -> 36.2 | Improved |
| Enable tool use for agents | Gemini 3 Pro · BizBench final score | final score (%): 44.8 -> 41.8 | Hurt |
Closest papers by meaning
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.88)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.86)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.86)
- Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.85)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.85)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
