Research map / Large language models / Retrieval and knowledge

Bian Que: An Agentic Framework with Flexible Skill Arrangement for Online System Operations

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Bian Que is an agentic O&M framework for online engine systems that uses flexible, LLM-generated and human-correctable Skills to select data and knowledge per event, with a unified self-evolving feedback loop, deployed on Kuaishou e-commerce search.

Problem

LLM agents for operating large-scale online engine systems are bottlenecked by orchestration: selecting the relevant data (metrics, logs, change events) and knowledge (handbook rules, practitioner experience) for each operational event, under frequent releases and undocumented knowledge.

Approach

Abstracts O&M into three canonical agent patterns (release interception, proactive inspection, alert RCA) and encodes event-to-(data, knowledge) mappings as LLM-generated, natural-language-updatable Skills; a single feedback signal drives both memory-to-knowledge distillation and Skill refinement.

Key results

• Online deployment over six months reduced fired alerts by 75% and achieved 80% RCA accuracy, with MTTR cut by over 50%.
• Offline on 104 curated events, end-to-end pass@1 rose from 94.2% (initialization pass@5) to 97.1% after one human correction round, reaching 99.0% after three rounds.
• Ablation: static Skills reduced pass@5 from 94.2% to 83.7%, and disabling knowledge retrieval reduced it to 86.5%.

Surprising findings

• Qwen3.5-35B-FP8 reached roughly the same RCA pass@1 as frontier models (GLM5, DeepSeek-V3.2), suggesting model scale above ~30B matters more than frontier status.
• Without feedback, online accuracy declined from about 75% to 32% over 13 days as traffic drifted.

Limitations

• Does not automate remediation actions such as rollbacks or scaling.
• Keyword-based Skill matching may fail on novel event types.
• Residual RCA errors concentrate on newly deployed services with sparse knowledge; compounding effect of the two pathways is not quantified.

Experimental setup

Datasets: production operational events from Kuaishou e-commerce search engine logs

Benchmarks: 104-event curated operational dataset (44 alerts, 60 inspections) 36-event alert RCA subset

Metrics: pass@k alert volume reduction RCA accuracy MTTR non-actionable alert ratio

Baselines: Static Skills NoKnow

Builds on: RAG ReAct

Closest papers by meaning

• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.85)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.85)
• EcoGEO: Trajectory-Aware Evidence Ecosystems for Web-Enabled LLM Search Agents (NeurIPS 2026 · similarity 0.85)
• AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation (ICML 2026 · similarity 0.84)
• WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.84)
• Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.84)