Research map / Multi-agent systems and games / LLM agents

CyberJurors: A Multi-Agent Simulation Task for E-Commerce Disputes Verdict

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the E-commerce Dispute Verdict (EDV) task, the 6,000-case multimodal VerdictBench, and CyberJurors, a 17-juror multi-agent framework with four-stage individual reasoning and multi-round jury consensus guided by verdict precedents.

Problem

Automating crowdsourced e-commerce dispute verdicts requires grounding pivotal clues from redundant, multi-round, multimodal evidence under flexible platform-specific rules, which existing legal and court-simulation methods handle poorly.

Approach

Combine an individual four-stage verdict chain-of-thought with iterative select-perceive clue grounding, and a jury-level discussion among 17 heterogeneous jurors guided by a retrieved verdict precedent base to reduce individual bias and align with real jury votes.

Key results

• CyberJurors reaches 0.7292 accuracy on VerdictBench, versus 0.6673 for AgentCourt and 0.6354 for Gemini-3-Pro (best single model).
• Ablation: baseline 0.5416 accuracy rises to 0.7252 with IV-CoT, jury simulation and verdict precedents added incrementally.
• Average MLLM accuracy (52.98%) is reported as lower than average text-only LLM accuracy (55.78%).

Surprising findings

• Current MLLMs underperform text-only LLMs on average, which the authors attribute to failure to perceive fine-grained visual clues in long contexts.
• Sellers win 62.6% of cases in the dataset, and evidence volume does not predict verdict outcome.

Limitations

• Full 17-juror simulation consumes very large token budgets (about 62M tokens per 100 cases).
• Evaluation is restricted to a binary buyer/seller verdict and does not yet produce compromise plans.
• Text-only court simulator baselines ignore visual evidence, so the multimodal comparison is not like-for-like.

Experimental setup

Datasets: VerdictBench

Benchmarks: VerdictBench

Metrics: Accuracy Weighted F1 Macro F1 Macro Recall Macro Precision MAE RMSE token consumption

Baselines: DeepSeek-V3 Qwen-Plus GPT-5.2-Chat GLM-4 Dolphin3.0-R1-24B Llama-2-13B Gemini-3-Pro Gemini-2.5-Flash-Lite Doubao-Seed-1.6 Claude-Opus-4.5 GPT-5.2 Grok-4

Builds on: ChatEval AgentCourt Chain-of-Thought

Closest papers by meaning

• Simulating Dispute Mediation with LLM-Based Agents for Legal Research (AAAI 2026 · similarity 0.84)
• Multi-Agent Debate for LLM Judges with Adaptive Stability Detection (NeurIPS 2025 · similarity 0.82)
• RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.81)
• Collaborative Disagreement Resolution for Scalable Oversight (ICML 2026 · similarity 0.81)
• JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.81)
• When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.80)