Research map / Large language models / Retrieval and knowledge
CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CyberGen, a three-step prompt-chaining method that uses AISecKG knowledge-graph triples to prompt ChatGPT for cybersecurity QA pairs, releases the expert-validated CyberQ dataset (~4k pairs), and fine-tunes T5-small/base on it.
Problem
Cybersecurity education lacks publicly available question-answer datasets, and manually creating and validating QA pairs from course material is slow and costly, while LLMs tend to hallucinate in this domain.
Approach
Ground LLM question-answer generation in knowledge-graph triples from a cybersecurity ontology through chained zero-shot, few-shot, and ontology-driven prompts, with domain-expert validation of the outputs.
Key results
- Domain experts found over 95% of the ChatGPT-generated cybersecurity question-answer pairs factually correct.
- T5-base outperformed T5-small on almost all metrics, with the best results on the Zero-Shot subset (ROUGE-1 0.43, METEOR 0.29 for T5-base on ZS).
- The CyberQ dataset contains about 4k QA pairs (1027 ZS, 332 FS, 2171 OD questions).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use T5-B instead of T5-S | Cybersecurity QA model trained on CyberQ Ontology-Driven data · CyberQ Ontology-Driven | BLEU: 0.10 -> 0.12 | Helped |
| Use T5-B instead of T5-S | Cybersecurity QA model trained on CyberQ Few-Shot data · CyberQ Few-Shot | ROUGE-1: 0.32 -> 0.33 | Helped |
| Use T5-B instead of T5-S | Cybersecurity QA model trained on CyberQ Zero-Shot data · CyberQ Zero-Shot | BLEU: 0.17 -> 0.18 | Helped |
| Train on Zero-Shot rather than Ontology-Driven CyberQ data | T5-S cybersecurity QA model · CyberQ | BLEU: 0.10 -> 0.17 | Helped |
| Train on Zero-Shot rather than Few-Shot CyberQ data | T5-S cybersecurity QA model · CyberQ | BLEU: 0.04 -> 0.17 | Helped |
| Train on Zero-Shot rather than Ontology-Driven CyberQ data | T5-B cybersecurity QA model · CyberQ | BLEU: 0.12 -> 0.18 | Helped |
| Train on Zero-Shot rather than Few-Shot CyberQ data | T5-B cybersecurity QA model · CyberQ | BLEU: 0.04 -> 0.18 | Helped |
Surprising findings
- Few-shot and ontology-driven prompts showed more hallucination and repetition when additional context was added.
- ChatGPT gave inconsistent answers to similar questions, for example on whether Windows is secure.
Limitations
- Expert validation covered only random samples, and questions on attack analysis and mitigation were sometimes answered incorrectly.
- ChatGPT showed inconsistency, repetition, and biased or unreliable responses.
- Evaluation of the QA model relies on n-gram overlap metrics against ChatGPT-generated gold answers, with no human assessment of the trained model.
Experimental setup
Datasets: AISecKG CyberQ
Benchmarks: CyberQ
Metrics: BLEU ROUGE-1 ROUGE-2 ROUGE-L ROUGE-LSum METEOR Flesch-Kincaid Grade Level Gunning Fog token-type ratio
Baselines: ChatGPT zero-shot prompting ChatGPT few-shot prompting
Builds on: AISecKG chain-of-thought prompting
Closest papers by meaning
- CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions (AAAI 2025 · similarity 0.84)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.83)
- KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering (NeurIPS 2026 · similarity 0.83)
- Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text (ICML 2026 · similarity 0.83)
- Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms (AAAI 2025 · similarity 0.82)
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.