Research map / Large language models / Agents, code and math
CVE-LLM: Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an MPT-7B-based LLM (CVE-LLM) via domain-adaptive pretraining and instruction SFT on historical medical-device vulnerability evaluations, enriched with CAPEC/CWE knowledge graph context, and deploys it human-in-the-loop at Siemens Healthineers.
Problem
Manual vulnerability evaluation of CVE notifications across a large medical device portfolio is slow and expert-dependent, and LLMs have seen little real-world study for automating these evaluations.
Approach
Fine-tune a domain-adapted decoder LLM on expert-written historical evaluations of asset-notification pairs, enriched with CAPEC prerequisites, severities and mitigations pulled from a cybersecurity knowledge graph (SEPSES/CSKG), so new vulnerabilities can be handled without retraining.
Key results
- CVE-LLM reaches micro-F1 0.94 on VEXCategory, 0.90 on VEXJustification, and 0.96 on Vector on the test set, outperforming Mistral-7B and Llama2-7B fine-tuned baselines (e.g., 0.64 and 0.56 on VEXCategory).
- Ablation on the test set shows removing DAPT drops VEXCategory from 0.94 to 0.83 and Vector from 0.96 to 0.84, while removing the ontology has little effect on the test set (0.93 VEXCategory).
- Inference is about 1 second per evaluation versus a mean of 194 seconds for an expert, with vLLM giving roughly 10x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CVE-LLM instead of Mistral-7B | Vulnerability evaluation · Test dataset, VEXCategory | micro-F1: 0.64 -> 0.94 | Helped |
| Use CVE-LLM instead of Mistral-7B | Vulnerability evaluation · Test dataset, VEXJustification | micro-F1: 0.60 -> 0.90 | Helped |
| Use CVE-LLM instead of Mistral-7B | Vulnerability evaluation · Test dataset, InternalComment | Rouge-L: 0.71 -> 0.79 | Helped |
| Use CVE-LLM instead of Mistral-7B | Vulnerability evaluation · Test dataset, CustomerComment | Rouge-L: 0.70 -> 0.88 | Helped |
| Use CVE-LLM instead of Mistral-7B | Vulnerability evaluation · Test dataset, Vector | micro-F1: 0.59 -> 0.96 | Helped |
| Add domain-adaptive pretraining (ablation) | CVE-LLM · Test dataset, VEXCategory | micro-F1: 0.83 -> 0.94 | Helped |
| Add domain-adaptive pretraining (ablation) | CVE-LLM · Test dataset, VEXJustification | micro-F1: 0.80 -> 0.90 | Helped |
| Add domain-adaptive pretraining (ablation) | CVE-LLM · Test dataset, InternalComment | Rouge-L: 0.64 -> 0.79 | Helped |
| Add domain-adaptive pretraining (ablation) | CVE-LLM · Test dataset, CustomerComment | Rouge-L: 0.69 -> 0.88 | Helped |
| Add domain-adaptive pretraining (ablation) | CVE-LLM · Test dataset, Vector | micro-F1: 0.84 -> 0.96 | Helped |
| Add ontology enrichment (ablation) | CVE-LLM · Test dataset, InternalComment | Rouge-L: 0.80 -> 0.79 | No clear effect |
| Vary temperature from 0 to 1 (ablation) | CVE-LLM-Eval inference · Test dataset | model performance | No clear effect |
Surprising findings
- Ontology enrichment did not substantially improve test-set performance but showed a significant impact during deployment on unseen notifications and assets.
- Changing sampling temperature and top-p produced no observable change in outputs, and performance declined beyond beam size 7.
Limitations
- Hallucination of named entities such as software versions and component names.
- Performance degrades on long multi-vulnerability instructions, which make up about 10% of the test set.
- Data and model are proprietary to a single manufacturer's portfolio, limiting generalizability.
Experimental setup
Datasets: NVD CVE descriptions SHS organization-wide vulnerability documents SHS Assets, Notifications, and Evaluations datasets CAPEC entries via SEPSES knowledge graph
Benchmarks: SHS held-out test set (44K evaluations) SHS post-deployment production set (10K evaluations)
Metrics: ROUGE-L micro-F1
Baselines: Llama3-RAG Mistral-7B Llama2-7B
Builds on: MPT-7B Alpaca vLLM SEPSES cybersecurity knowledge graph
Closest papers by meaning
- CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions (AAAI 2025 · similarity 0.83)
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.83)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.82)
- Position: Editing Large Language Models Poses Serious Safety Risks (ICML 2025 · similarity 0.82)
- A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle (NeurIPS 2026 · similarity 0.82)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.