Research map / Multi-agent systems and games / LLM agents

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SLS-Bench, a 90-task benchmark for end-to-end incident log summarization built by a report-grounded synthetic pipeline (incident models, simulators, generator-verifier loops), and evaluates 14 LLM agents with an LLM judge.

Problem

Existing log benchmarks use short log windows and granular outputs, so end-to-end summarization of large log streams into incident reports is not evaluated; collecting real logs and expert summaries at scale is prohibitively expensive.

Approach

Convert publicly available postmortem reports into structured incident models (system structure, log emission mechanisms, phased timeline), then generate deterministic log simulators and reference summaries from them, with generator-verifier loops and statistical comparison to Loghub-2.0 logs.

Key results

• GPT-5.4 (medium) and (high) reach 0.73 mean score, at $1.56-$2.19 and 6.9-9.5 minutes per problem.
• Open-weight models lag substantially; best open-weight model GLM-5 scores 0.49, while Nemotron 3 Nano scores 0.06.
• Judge calibration accuracy is 90% for GPT-5.2 (medium) and 62% for Sonnet 4.6 (low); the two judges' model rankings correlate at Spearman 0.98.

Surprising findings

• Within a size class, lower reasoning effort often gives the best cost and time efficiency ratios.
• 21% of Python tool outputs in GPT-5.4 trajectories were precomputable.

Limitations

• Coverage limited to 90 publicly available incident reports.
• Incident model caps complexity; logs lack background noise of full production environments.
• Logs and reference summaries come from the same incident model, so evaluation depends on references faithfully reflecting log evidence; no direct validation on real incident logs.

Experimental setup

Datasets: Loghub-2.0 publicly available incident reports (343 collected, 289 unique, 90 selected)

Benchmarks: SLS-Bench

Metrics: score (0-1 mapped from five rubric labels) cost per problem (USD) time per problem (min) score per dollar score per minute

Baselines: GPT-5.4 Mini GPT-5.4 Nano Opus 4.6 Sonnet 4.6 Haiku 4.5 GLM-5 MiniMax-M2 Nemotron 3 Super DeepSeek V3.2 Mistral Large 3 Ministral 3 Qwen3-Next

Builds on: Loghub-2.0 pandas

Closest papers by meaning

• Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations (NeurIPS 2026 · similarity 0.83)
• AIR: Improving Agent Safety through Incident Response (ICML 2026 · similarity 0.82)
• AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.82)
• OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories (NeurIPS 2026 · similarity 0.82)
• CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.81)
• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.81)