Research map / Label-efficient and robust learning / Knowledge distillation

Fine-Grained Distillation for Long Document Retrieval

AAAI 2024 · Label-efficient and robust learning › Knowledge distillation · approach: Self and data-free distillation · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes fine-grained distillation (FGD) that distills passage- and sentence-level relevance from a cross-encoder into a single-vector long-document bi-encoder using global-consistent granularity embeddings, reaching SOTA on MS-MARCO Doc and TREC 2019 Doc.

Problem

Knowledge distillation from cross-encoders to bi-encoders is less effective for long document retrieval because documents cover multiple topics, causing a granularity mismatch between query-dependent cross-encoder scores and query-agnostic document-level bi-encoder vectors.

Approach

Produce single-vector-consistent embeddings for passages and sentences via global [CLS] attention pooling, distill multi-granular score distributions from a cross-encoder at fine granularity only during training, and mine hierarchical hard negatives across granularities; inference remains single-vector.

Key results

• FGD achieves 0.440 MRR@100 on MS-MARCO doc dev, versus 0.422 for the previous SoTA COSTA.
• FGD reaches 0.635 nDCG@10 and 0.349 Recall@100 on TREC 2019 Doc, the best reported in Table 1.
• Multi-vector retrieval (FGD + multi) raises MRR@100 to 0.444 on MS-MARCO dev.

Measured effects reported in the paper

Surprising findings

• Brute-force document-level distillation yields only about 0.1% gain on long document retrieval after extensive tuning, versus over 1% in passage retrieval.
• Adding document-level distillation to FGD lowers MRR@100 by 0.4%, which the authors attribute to granularity mismatch.

Limitations

• Gain over strong pretrained baselines is modest on TREC 2019 and the multi-vector variant departs from pure single-vector retrieval.
• Fine-grained distillation relies on weak labels, since sub-document positives are assumed to be all passages/sentences in the gold document.

Experimental setup

Datasets: MS-MARCO document retrieval MS-MARCO passage retrieval

Benchmarks: MS-MARCO document dev TREC 2019 Deep Learning document

Metrics: MRR@100 Recall@100 nDCG@10

Baselines: BM25 DeepCT ANCE BERT (Ma et al. 2022) STAR ICT PROP B-PROP SEED RepCONC JPQ ADORE+STAR

Builds on: ED-MLM / SimLM coCondenser STAR ADORE COSTA

Closest papers by meaning

• Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe (NeurIPS 2025 · similarity 0.86)
• Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.85)
• Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization (ICLR 2024 · similarity 0.85)
• Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.84)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.84)
• Distribution-Driven Dense Retrieval: Modeling Many-to-One Query-Document Relationship (AAAI 2025 · similarity 0.84)