Research map / Vision-language and multimodal / CLIP and zero-shot
Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-Hoc Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Wikiformer pre-trains a BERT-base retriever on four pseudo query-document tasks built from Wikipedia's structure (heading trees, titles/abstracts, See Also graph) for ad-hoc retrieval.
Problem
Pre-trained language models for ad-hoc IR use plain Wikipedia text, leaving the rich structured information of Wikipedia (titles, abstracts, heading hierarchy, See Also links) unexploited for pre-training.
Approach
Four pre-training objectives (Simulated Re-ranking, Representative Words Identification, Abstract Texts Identification, Long Texts Matching) that sample pseudo query-document pairs from Wikipedia's heading tree, title-abstract pairs, and a directed See Also graph, trained with softmax cross-entropy and summed losses.
Key results
- Zero-shot on MS MARCO, Wikiformer reaches MRR@10 0.2844 vs 0.2736 for ARES and 0.2656 for BM25.
- Fine-tuned on MS MARCO, Wikiformer reaches MRR@10 0.4085 vs 0.3995 for the best prior pre-trained baseline (ARES).
- On TREC Covid rnd2, zero-shot nDCG@10 is 0.5449 vs 0.4993 for the best prior PLM (PROP_MS and ARES); on LeCaRD fine-tuned nDCG@5 is 0.7722 vs 0.7553 for BERT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with Wikiformer's four Wikipedia-structure tasks instead of BERT-base alone (ablation) | BERT-base · MS MARCO zero-shot | MRR@10: 0.1684 -> 0.2844 | Helped |
| add SRR (ablation) | Wikiformer · MS MARCO zero-shot | MRR@10: 0.2334 -> 0.2844 | Helped |
| add RWI (ablation) | Wikiformer · MS MARCO zero-shot | MRR@10: 0.2596 -> 0.2844 | Helped |
| add ATI (ablation) | Wikiformer · MS MARCO zero-shot | MRR@10: 0.2641 -> 0.2844 | Helped |
| add LTM (ablation) | Wikiformer · MS MARCO zero-shot | MRR@10: 0.2726 -> 0.2844 | Helped |
| add SRR (ablation) | Wikiformer · MS MARCO zero-shot | MRR@100: 0.2441 -> 0.2911 | Helped |
| add RWI (ablation) | Wikiformer · LeCaRD fine-tuned | nDCG@5: 0.7685 -> 0.7722 | Helped |
| add ATI (ablation) | Wikiformer · LeCaRD fine-tuned | nDCG@5: 0.7627 -> 0.7722 | Helped |
| add LTM (ablation) | Wikiformer · LeCaRD fine-tuned | nDCG@5: 0.7574 -> 0.7722 | Helped |
| use Wikiformer pre-training rather than ARES | BERT-base retrieval model · TREC Covid rnd2 zero-shot | nDCG@10: 0.4993 -> 0.5449 | Helped |
Surprising findings
- Wikiformer surpasses the Query Likelihood model using only 100 pseudo query-document pairs from the SRR task.
- Neural IR models (KNRM, Conv-KNRM) still underperform BM25 and QL even after fine-tuning.
Limitations
- Baselines HARP and Webformer were reproduced by the authors since no checkpoints were released.
- Chinese legal datasets lack a Chinese IR-specific pre-trained baseline; only BERT is compared there.
- Pre-training relies on English Wikipedia structure, which may not transfer to corpora lacking such structure.
Experimental setup
Datasets: English Wikipedia 20220101 MS MARCO TREC DL 2019 TREC Covid Round2 LeCaRD CAIL-LCR
Benchmarks: MS MARCO TREC DL 2019 TREC Covid Round2 LeCaRD CAIL-LCR
Metrics: MRR@10 MRR@100 nDCG@10 nDCG@100 P@5 nDCG@5 nDCG@15
Baselines: BM25 Query Likelihood KNRM Conv-KNRM BERT PROP_MS PROP_WIKI HARP ARES Webformer
Builds on: BERT PROP HARP ARES Webformer
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.85)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.84)
- Learning Pattern-Based Extractors from Natural Language and Knowledge Graphs: Applying Large Language Models to Wikipedia and Linked Open Data (AAAI 2024 · similarity 0.83)
- WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia (AAAI 2024 · similarity 0.83)
- BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models (ICLR 2024 · similarity 0.82)
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.