Research map / Label-efficient and robust learning / Recognition and long-tail

A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Dense features and foundation priors

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HackerSignal, a 7.45M-document multi-source CTI corpus linking hacker-forum discourse, exploit archives, advisories and fix commits via CVE IDs, with three temporal benchmark tasks and a baseline ladder.

Problem

Public datasets rarely link hacker-community discourse to the CVE vulnerability lifecycle (exploit, advisory, fix), and existing CTI benchmarks lack temporal OOD splits and provenance governance.

Approach

A governed, temporally split multi-source corpus unified by a shared CVE identifier space, with three benchmark tasks (CVE linkage retrieval, 8-class exploit type classification, CVE-disjoint temporal generalization) and a heterogeneous baseline ladder plus leakage and audit diagnostics.

Key results

• On CVE linkage retrieval (Task 1, 1,990 queries over 340K NVD corpus), E5-base-v2 reaches MRR 0.587 and R@10 0.769, outperforming BM25 (MRR 0.528) and the hybrid BM25+mpnet (MRR 0.567).
• On 8-class exploit type classification (Task 2), BiLSTM reaches macro-F1 0.871, above SecBERT (0.846) and the best bag-of-words SVM (0.799).
• On CVE-disjoint temporal generalization (Task 3), E5-base-v2 attains MRR 0.602 and R@1 0.509, the best among the evaluated retrievers.

Surprising findings

• Domain-specific SecBERT underperforms a bidirectional BiLSTM on exploit type classification, suggesting domain pre-training gives only modest benefit here.
• The vanilla RNN collapses to macro-F1 0.142, far below GRU/LSTM (0.826).

Limitations

• CVE linkages are metadata-derived (URLs, regex extraction) rather than expert-adjudicated, so some false linkages may exist.
• Cross-lingual retrieval and multilingual baselines are not evaluated; corpus is predominantly English.
• No LLM rerankers or retrieval-augmented generation baselines are evaluated.

Experimental setup

Datasets: NVD ExploitDB 0day.today Packet Storm GitHub Advisory CISA KEV HackerOne CVEfixes Gayanku HackForums DeepDarkCTI CrackingArena

Benchmarks: HackerSignal CVE-R HackerSignal ETC HackerSignal TG

Metrics: Recall@1 Recall@5 Recall@10 MRR macro-F1 per-class F1

Baselines: BM25 MiniLM-L6-v2 all-mpnet-base-v2 BGE-base-v1.5 E5-base-v2 Hybrid BM25+mpnet Decision Tree TF-IDF+LR SVM RNN GRU LSTM

Builds on: BEIR WILDS WRENCH KILT CVEfixes

Closest papers by meaning

• CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities (ICML 2026 · similarity 0.86)
• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.86)
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.85)
• HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities (ICLR 2026 · similarity 0.85)
• CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.85)
• KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards (NeurIPS 2026 · similarity 0.84)