Research map / Label-efficient and robust learning / Self-supervised and contrastive
Unsupervised Pretraining for Fact Verification by Language Model Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SFAVEL, an unsupervised pretraining framework that distills frozen language-model features into a knowledge-graph-based claim-fact matching model using a distillation loss, a contrastive loss, and a scoring loss, evaluated on FEVER and FB15k-237.
Problem
Fact verification requires claim and evidence features that are semantically meaningful and aligned, but existing methods depend on annotated claim-label corpora; the paper targets annotation-free claim-fact alignment for fact verification over knowledge graphs.
Approach
Distill features from a frozen pre-trained language model (teacher) into a relational graph attention knowledge model (student) using a normalized feature distillation loss, combined with a claim-conditioned contrastive loss over positive and negative facts (in-batch and in-KB negatives) and a margin-based scoring loss, with no annotations.
Key results
- On FEVER test with a linear probe, SFAVEL reaches 89.48 label accuracy and 86.15 FEVER score, versus 79.47/76.82 for the best supervised ProoFVer.
- On FB15k-237 test, SFAVEL reports Hits@1 34.5, Hits@3 45.6, Hits@10 64.1, a +5.3% Hits@1 gain over MoCoSA.
- Ablation over backbones shows GPT-2-XL is the best feature extractor, and even T5-Small (60M) slightly exceeds the prior state of the art (+0.54% accuracy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SFAVEL instead of ProoFVer | FEVER test · FEVER test | label accuracy: +9.58% | Helped |
| use SFAVEL instead of the previous state of the art | FB15k-237 · FB15k-237 test | Hits@1: +5.3% | Helped |
| use SFAVEL instead of the previous state of the art | FB15k-237 · FB15k-237 test | Hits@3: +3.6% | Helped |
| use SFAVEL instead of the previous state of the art | FB15k-237 · FB15k-237 test | Hits@10: +6.3% | Helped |
| replace GPT-2-XL backbone with T5-Small (ablation) | SFAVEL · FEVER dev | label accuracy | Hurt |
| set fact-selection K to 1 instead of 5 (ablation) | SFAVEL · FEVER dev | label accuracy | Hurt |
| set fact-selection K to 10, 20, or 30 instead of 5 (ablation) | SFAVEL · FEVER dev | label accuracy | Hurt |
| add claim-fact distillation loss (ablation) | SFAVEL · FEVER dev | label accuracy | Helped |
| add intra-sample contrastive loss (ablation) | SFAVEL · FEVER dev | label accuracy | Helped |
| add scoring loss (ablation) | SFAVEL · FEVER dev | label accuracy | Helped |
Surprising findings
- Even the 60M-parameter T5-Small backbone already surpasses prior state-of-the-art accuracy on FEVER.
Limitations
- Relies on a frozen backbone and a linear probe; the downstream fine-tuning adaptation is optional and not extensively explored.
- FEVER comparison relies on a knowledge graph built from Wikidata5m and verbalized triples rather than the original Wikipedia text evidence.
- Ablations are run only on the FEVER dev split with GPT-2-XL as backbone.
Experimental setup
Datasets: FEVER Wikidata5m FB15k-237
Benchmarks: FEVER FB15k-237
Metrics: label accuracy FEVER score Hits@1 Hits@3 Hits@10
Baselines: GEAR KGAT GERE CorefBERT DREAM ProoFVer LaPraDoR KG-BERT StAR KG-S2S SimKGC MoCoSA
Builds on: MoCo RGAT feature-based knowledge distillation
Closest papers by meaning
- Enhancing Multi-Hop Fact Verification with Structured Knowledge-Augmented Large Language Models (AAAI 2025 · similarity 0.87)
- Beyond Text: Fine-Grained Multi-Modal Fact Verification with Hypergraph Transformers (AAAI 2025 · similarity 0.85)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.85)
- Truth is Universal: Robust Detection of Lies in LLMs (NeurIPS 2024 · similarity 0.84)
- Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables (AAAI 2024 · similarity 0.84)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.