Research map / Label-efficient and robust learning / OOD and anomaly detection
LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LogFormer, a transformer-based log anomaly detector that pre-trains on a source log domain (BGL) with a Log-Attention module and then transfers to target domains via parallel adapters with frozen encoder weights.
Problem
Log anomaly detection models generalize poorly across log domains and must be retrained for each new domain; log parsing also discards parameter information that may be anomaly-relevant.
Approach
A two-stage pre-train then adapter-tuning pipeline for log anomaly detection, plus Log-Attention that adds learnable biases derived from parsed parameter embeddings to self-attention, so that only a small number of adapter parameters are trained on the target domain.
Key results
- LogFormer reaches F1 0.98 on HDFS and 0.99 on Thunderbird, versus 0.91 for the best baseline (LogRobust) on HDFS.
- Adapter-based tuning uses about 3.5%-5.5% of the trainable parameters of direct fine-tuning, with F1 about 1% higher on average.
- Log-Attention improves average F1 by 3.6% over standard self-attention across HDFS, BGL and Thunderbird.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add source-domain pre-training before target-domain tuning (ablation) | LogFormer on BGL · BGL | F1 | Helped |
| add adapter-based tuning instead of directly fine-tuning the pretrained model (ablation) | LogFormer · HDFS and Thunderbird | F1: HDFS: 0.962 -> 0.974 (2 layers); Thunderbird: 0.981 -> 0.987 (2 layers) | Helped |
| add Log-Attention to encode information removed by parsing (ablation) | LogFormer_S · HDFS, BGL, and Thunderbird | F1: HDFS: 0.911 -> 0.952; BGL: 0.943 -> 0.974; Thunderbird: 0.935 -> 0.962 | Helped |
| increase adapter-tuning encoder layers from 1 to 4 (ablation) | LogFormer · HDFS and Thunderbird | F1: HDFS: 0.957 -> 0.981; Thunderbird: 0.972 -> 0.998 | Helped |
| increase direct fine-tuning encoder layers from 2 to 4 (ablation) | LogFormer_P · HDFS | F1: 0.962 -> 0.961 | No clear effect |
| increase direct fine-tuning encoder layers from 2 to 4 (ablation) | LogFormer_P · Thunderbird | F1: 0.981 -> 0.980 | No clear effect |
| replace the parallel adapter with a serial adapter (ablation) | LogFormer · HDFS, BGL, and Thunderbird | F1: HDFS: 0.980 -> 0.982; BGL: 0.969 -> 0.971; Thunderbird: 0.988 -> 0.992 | Mixed |
| replace the parallel adapter with LoRA (ablation) | LogFormer · HDFS, BGL, and Thunderbird | F1: HDFS: 0.980 -> 0.981; BGL: 0.969 -> 0.972; Thunderbird: 0.988 -> 0.993 | Mixed |
| use adapter-based tuning rather than training from scratch (ablation) | LogFormer on Thunderbird with 5k training examples · Thunderbird low-resource setting | F1: 34% improvements | Helped |
| use BGL rather than Thunderbird as the pre-training source (ablation) | LogFormer · HDFS and Thunderbird target domains | convergence speed | Helped |
| add Chain-of-Thought prompting to ChatGPT (ablation) | ChatGPT · HDFS, BGL, and Thunderbird | F1: HDFS: 0.78 -> 0.85; BGL: 0.74 -> 0.83; Thunderbird: 0.81 -> 0.88 | Helped |
Surprising findings
- ChatGPT, even with a two-hop chain-of-thought prompt, reaches only 0.78-0.88 F1, well below the trained models.
- Pre-training on BGL was found to be a better source domain than Thunderbird, based on loss-curve convergence rather than F1.
Limitations
- Source and target domain pairs are limited to a few public log datasets, and the BGL-as-source choice is justified heuristically.
- Baselines are trained on the union of source and target data, and the comparison to ChatGPT uses a single prompting setup.
Experimental setup
Datasets: HDFS BGL Thunderbird GAIA
Benchmarks: HDFS BGL Thunderbird GAIA
Metrics: Precision Recall F1 score training time testing time
Baselines: SVM DeepLog LogAnomaly LogRobust PLELog ChatGPT
Builds on: Sentence-BERT Drain Parallel adapter LoRA DeepLog
Closest papers by meaning
- ICAD-LLM: One-for-All Anomaly Detection via In-Context Learning with Large Language Models (AAAI 2026 · similarity 0.85)
- Towards Training-free Anomaly Detection with Vision and Language Foundation Models (CVPR 2025 · similarity 0.85)
- MPFM: Cross Multi-Domain Prototype Flow Matching for Log Anomaly Detection (ICML 2026 · similarity 0.84)
- Co-Progression Knowledge Distillation with Knowledge Prototype for Industrial Anomaly Detection (AAAI 2025 · similarity 0.82)
- AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization (AAAI 2026 · similarity 0.82)
- ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.