Research map / Multi-agent systems and games / LLM agents
Judgment Operators: A Composition-Invariant Substrate for Multi-Agent Action Spaces
Jun Li
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes judgment operators, a composition-invariant substrate for governing action spaces in heterogeneous multi-agent LLM systems, motivated by the failure of internal correction methods to enforce execution-time constraints.
Problem
Knowledge and governance fragment across agents in heterogeneous LLM multi-agent systems, causing composition-dependent behavior and violations that scale linearly with the number of agents.
Approach
A composition-invariant substrate called judgment operators that enforces execution-time constraints on agent action spaces, rather than relying on internal self-correction (Reflexion, CRITIC, Self-Refine), which the authors observe fails to enforce such constraints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add precedent-based repair learning (JO-A over JO-static) (ablation) | JO-static · Overcooked (N=200) | Violation rate (%) and success (%): VR 2.6 -> 0.0; success 85.0 -> 100.0 | Improved |
| add external judgment operator JO-A to the agent | No Operator · Overcooked (N=200) | Violation rate (%): 56.2 -> 0.0 | Improved |
| embed all constraints in a centralized system prompt | No Operator · Overcooked (N=200) | Violation rate / success (%): VR 56.2 -> 7.1; success 10.0 -> 2.0 | Mixed |
| add chain-of-thought constraint checking | No Operator · Overcooked (N=200) | Violation rate / success (%): VR 56.2 -> 14.8; success 10.0 -> 0.0 | Mixed |
| add Self-Refine internal correction | No Operator · Overcooked (N=200) | Violation rate / success (%): VR 56.2 -> 20.9; success 10.0 -> 0.0 | Mixed |
| add Reflexion internal self-correction | No Operator · Overcooked (N=200) | Violation rate (%): 56.2 -> 56.2 | No clear effect |
| add CRITIC internal correction | No Operator · Overcooked (N=200) | Violation rate (%): 56.2 -> 64.8 | Hurt |
| add LlamaGuard reject-only filtering | No Operator · Overcooked (N=200) | Violation rate / success (%): VR 56.2 -> 67.8; success 10.0 -> 15.0 | Mixed |
| add JO-A mediation to agents (agent count change K1 to K2) | Overcooked multi-agent · Composition sensitivity, Overcooked | Delta comp (pp): 15.1pp -> 0.1pp | Improved |
| add JO-A mediation to heterogeneous model mix | Protocol agents · Composition sensitivity, Protocol | Delta comp (pp): 51.0pp -> 6.0pp | Improved |
| increase bypass rate (partial mediation) from 0% to 30% (ablation) | JO-A · Overcooked partial mediation (N=100) | Violation rate (%): 0.0 -> 3.2 (bound 16.9) | Hurt |
| apply JO-A with judgment artifacts learned by GPT-5.2 to a student model zero-shot | GPT-4o-mini · WebArena-Wikipedia (N=200) | Success (%): 2.4 -> 22.9 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Reflexion — Reflexion: language agents with verbal reinforcement learning (NeurIPS 2023) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Surprising findings
- Internal correction methods (Reflexion, CRITIC, Self-Refine) fail to enforce execution-time constraints in multi-agent settings, according to the abstract's motivating observations.
Limitations
- Full text unavailable; evaluation details unknown
Experimental setup
Baselines: Reflexion CRITIC Self-Refine
Builds on: Reflexion CRITIC Self-Refine
Closest papers by meaning
- MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles (ICML 2025 · similarity 0.88)
- Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.88)
- PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (NeurIPS 2026 · similarity 0.88)
- RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code (ICLR 2025 · similarity 0.88)
- Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.88)
- MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
