Research map / Large language models / Language, safety and interpretability
OWL: A Large Language Model for IT Operations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Owl, a LLaMA2-13B-based LLM for IT operations, trained on the Owl-Instruct dataset and evaluated on the new Owl-Bench, with a training-free long-context method (HMCE) and a mixture-of-adapter LoRA tuning strategy.
Problem
General-purpose LLMs lack specialized knowledge for IT operations and maintenance (O&M), and existing context-length and tuning methods and benchmarks do not address this domain.
Approach
Builds a domain instruction dataset via expert seeds plus GPT-4-assisted self-instruct augmentation, extends NBCE to a homogeneous Markov chain formulation (HMCE) for training-free long-context inference, and uses a top-k mixture of LoRA adapters gated by language representations for multi-domain tuning.
Key results
- On Owl-Bench Q&A, Owl-13b achieves an average GPT-4-judged score of 8.86, above LLaMA2-13b (8.57) and Qwen-7b (8.41).
- On zero-shot log anomaly detection on BGL and Spirit, Owl reaches F1 of 0.446 and 0.518, versus 0.384 and 0.450 for LogPrompt (ChatGPT), an average 6.5% F1 gain.
- HMCE keeps perplexity at 6.10 at 8192 tokens, versus 314.49 for the base Owl-13b and 8.24 for NBCE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply HMCE context extension | Owl-13b · Owl-Bench long-context inference, sequence length 8192 | perplexity: 314.49 -> 6.10 | Helped |
| replace NBCE with HMCE context extension | Owl-13b · Owl-Bench long-context inference, sequence length 8192 | perplexity: 8.24 -> 6.10 | Helped |
| add Mixture-of-Adapter (ablation) | Owl · Owl-Bench multiple-choice, zero-shot | mean accuracy: 0.73 -> 0.76 | Helped |
| replace Mixture-of-Adapter with LoRA fine-tuning (ablation) | Owl · Owl-Bench multiple-choice, zero-shot | mean accuracy: 0.76 -> 0.71 | Hurt |
| use Owl instead of LogPrompt (ChatGPT) | zero-shot log anomaly detection · BGL | F1-score: 0.384 -> 0.446 | Helped |
| use Owl instead of LogPrompt (ChatGPT) | zero-shot log anomaly detection · Spirit | F1-score: 0.450 -> 0.518 | Helped |
| use Owl instead of LogPrompt (ChatGPT) | zero-shot log parsing · LogHub average | RandIndex: 0.865 -> 0.886 | Mixed |
| use Owl instead of LogPrompt (ChatGPT) | zero-shot log parsing · LogHub average | fine-level F1-score: 0.881 -> 0.894 | Helped |
Surprising findings
- Owl-13b without context extension has perplexity rising sharply with length (314.49 at 8192 tokens), while NBCE-based extension is less effective than HMCE.
- Zero-shot log anomaly detection remains modest even for LLM-based methods, according to the authors.
Limitations
- Zero-shot log anomaly detection performance remains modest.
- Owl-Bench Q&A pairwise comparisons have many draws, limiting discrimination.
- Owl-Bench Q&A evaluation relies on GPT-4 scoring without human validation reported.
Experimental setup
Datasets: Owl-Instruct LogHub
Benchmarks: Owl-Bench LogHub
Metrics: average score (1-10) pairwise win/tie/loss accuracy perplexity F1-score precision recall RandIndex
Baselines: ChatGPT LLaMA2-13b ChatGLM2-6b ChatGLM-6b Qwen-7b InternLM-7b DeepLog LogAnomaly LogRobust LogPrompt LogPPT
Builds on: Self-Instruct NBCE LoRA Baize LLaMA2
Closest papers by meaning
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.85)
- Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.84)
- Boosting the Potential of Large Language Models with an Intelligent Information Assistant (NeurIPS 2024 · similarity 0.84)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.84)
- HYVE: Hybrid Views for LLM Context Engineering over Machine Data (NeurIPS 2026 · similarity 0.84)
- TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.