Research map / Large language models / Language, safety and interpretability
HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HLM-Cite, a two-stage hybrid workflow that curriculum-finetunes a GTE embedding model to retrieve likely core citations from up to 100K candidates, then reranks them with GPT-based LLM agents (analyzer, decider, guider) for core-citation prediction.
Problem
Predicting which previous papers a new paper will cite, distinguishing core citations from superficial citations and non-citations, over very large candidate sets that exceed LLM context length.
Approach
Defines core citations via local citation-network relations (cited papers that are also cited by follow-up papers of the query), then combines a curriculum-finetuned embedding retriever (classification then NeuralNDCG ranking) with an LLM agentic ranking workflow that analyzes logical relations between query and candidates using a one-shot CoT example.
Key results
- H-LM (GPT4o) reaches overall PREC@5 0.655 and NDCG@5 0.686 on 10K candidate sets, versus 0.563 PREC@5 for the best embedding baseline GTE-large-v1.5, with the paper reporting up to 17.6% improvement over SOTA.
- Ablations show removing the second curriculum stage drops overall PREC@5 from 0.598 to 0.588 at the retrieval stage, and removing the guider drops full-workflow PREC@5 from 0.644 to 0.595.
- Scaling candidate sets to 100K yields up to 18.5% relative improvement over top baselines according to the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the HLM-Cite two-stage retrieval and LLM ranking workflow instead of GTE-large-v1.5 | Core citation prediction · Overall (10K candidates) | PREC@5: 0.563 -> 0.655 | Helped |
| Use the HLM-Cite two-stage retrieval and LLM ranking workflow instead of GTE-large-v1.5 | Core citation prediction · Overall (10K candidates) | NDCG@5: 0.606 -> 0.686 | Helped |
| Add curriculum Stage 1, which trains core-versus-non-citation classification (ablation) | GTE-base retrieval module · Overall retrieval set | PREC@5: 0.596 -> 0.598 | Helped |
| Add curriculum Stage 2, which ranks core, superficial, and non-citations (ablation) | GTE-base retrieval module · Overall retrieval set | PREC@5: 0.588 -> 0.598 | Helped |
| Add Analyzer reasoning about logical relationships between papers (ablation) | HLM-Cite ranking workflow · Overall | PREC@5: 0.630 -> 0.644 | Helped |
| Add the Guider's one-shot example (ablation) | HLM-Cite ranking workflow · Overall | PREC@5: 0.595 -> 0.644 | Helped |
| Add both Analyzer and Guider (ablation) | HLM-Cite ranking workflow · Overall | PREC@5: 0.581 -> 0.644 | Helped |
| Replace one-shot with domain-specific few-shot examples (ablation) | HLM-Cite ranking workflow · Overall | PREC@5: 0.635 -> 0.633 | No clear effect |
| Increase retrieval size; performance rises initially, then drops when retrieval size is large enough (ablation) | HLM-Cite · Natural and social science | PREC@5 | Mixed |
| Use GPT-4o instead of GPT-3.5 for the agents (ablation) | HLM-Cite agentic ranking · Overall | PREC@5: 0.635 -> 0.655 | Helped |
| Replace GPT agents with lightweight open-source agents (ablation) | HLM-Cite agentic ranking · Overall | PREC@5: 0.635 -> 0.593 | Hurt |
Surprising findings
- Few-shot examples with domain-specific content give no significant gain over one-shot, suggesting the logical structure of CoT matters more than domain knowledge.
- Lightweight open-source LLMs (Llama3-8B/70B, Mixtral, ChatGLM2-6B) perform significantly worse than GPT models in the agentic ranking.
Limitations
- LLM illusion problem may yield unfaithful analysis
- GPT-4o evaluated on a 10% test subset due to API rate limits
- Relies on proprietary GPT APIs
Experimental setup
Datasets: Microsoft Academic Graph (MAG)
Benchmarks: MAG-based core citation prediction dataset (19 scientific fields)
Metrics: PREC@3 PREC@5 NDCG@3 NDCG@5
Baselines: Keywords overlap SciBERT METAG PATTON SciPATTON SPECTER SciNCL SciMult SPECTER-2.0 BERT GTE-large-v1.5 OpenAI-embedding-3
Builds on: GTE NeuralNDCG chain-of-thought prompting SPECTER SciNCL
Closest papers by meaning
- CiteME: Can Language Models Accurately Cite Scientific Claims? (NeurIPS 2024 · similarity 0.85)
- Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning (ICLR 2026 · similarity 0.84)
- COMPOSE: Composing Future Theorems from Citations and Formal Structure (NeurIPS 2026 · similarity 0.83)
- Chain-of-Thought Improves Text Generation with Citations in Large Language Models (AAAI 2024 · similarity 0.83)
- SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models (ICML 2025 · similarity 0.83)
- SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.