Research map / Large language models / Language, safety and interpretability
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| probe hidden states instead of prompting the small LM | Qwen3-1.7B · MATH Logicality (binary) | weighted F1 (%): 51.34 -> 83.67 | Improved |
| probe hidden states instead of prompting the small LM | Qwen3-1.7B · MATH Logicality (multiclass 1-5) | weighted F1 (%): 15.06 -> 59.05 | Improved |
| probe hidden states instead of prompting the small LM | Llama-3.1-8B-Instruct · MATH Informativeness (binary) | weighted F1 (%): 86.67 -> 88.12 | Improved |
| use mean pooling instead of other pooling strategies (ablation) | INSPECTOR probe · MATH Informativeness (binary) | F1 | Improved |
| use logistic regression instead of other classifiers (ablation) | INSPECTOR probe · MATH Informativeness (binary) | F1 | Improved |
| use a larger small LM (Llama-3.1-8B vs Qwen3-1.7B) (ablation) | INSPECTOR probe · reasoning benchmarks | weighted F1 (%): 96.32 vs 92.65 (Llama-3.2-1B vs Llama-3.1-8B, MATH Fluency binary) | Mixed |
| train probe on one dataset and test on another (OOD) vs in-distribution (ablation) | INSPECTOR probe Qwen3-1.7B · GSM8K/MATH multiclass | weighted F1 (%): multiclass F1 drops to roughly 10-25% | Hurt |
| filter SFT data with probing classifier instead of random | Llama-2-7B-Chat distillation · reasoning benchmarks SFT | SFT performance | Improved |
Closest papers by meaning
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.90)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.89)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
