Research map / Large language models / Instruction and fine-tuning
From Words to Worth: Newborn Article Impact Prediction with LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLMs with LoRA on title and abstract to regress a same-period normalized citation index (TNCSI_SP) for newborn articles, releasing a ~12k-entry NAID dataset.
Problem
Predicting the future impact of newly published articles without external information (citations, venue, authors), since existing methods rely on historical or external data and citation counts are poorly comparable across fields and time.
Approach
Use the first generated token of a decoder-only LLM, passed through an MLP and sigmoid, to regress a normalized impact index (TNCSI_SP) computed from citations of concurrent papers within a 6-month window, so predictions use only title and abstract.
Key results
- Fine-tuned LLaMA-3-8B reaches MAE 0.216 and NDCG@20 0.901 on the NAID test set, outperforming MLP, LSTM, and ChatGPT-generated baselines in the title-abstract-only setting.
- Among tested LLMs, LLaMA-3 has the best NDCG (0.901) while Mistral (0.850) and Qwen-2 (0.774) trail; Phi-3 gets 0.742 with 3.8B parameters.
- Switching the regression target from TNCSI to TNCSI_SP improves the fine-tuned LLaMA-3 NDCG from 0.865 to 0.901.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-3 on TNCSI_SP using titles and abstracts | LLaMA-3 · NAID test set | NDCG: 0.674 -> 0.901 | Helped |
| Use TNCSI_SP instead of TNCSI as the regression target (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.865 -> 0.901 | Helped |
| Use TNCSI_SP instead of TNCSI as the regression target (ablation) | MLP-based · NAID test set | NDCG: 0.464 -> 0.634 | Helped |
| Use TNCSI_SP instead of TNCSI as the regression target (ablation) | LSTM-based · NAID test set | NDCG: 0.373 -> 0.646 | Helped |
| Use an application-area and key-technology prompt for key-phrase identification (ablation) | ChatGPT key-phrase generation · TKPD | NED: 0.30 -> 0.26 | Helped |
| Specify the normalized impact scale in the prediction prompt (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.889 -> 0.901 | Helped |
| Add open-access status to the title and abstract (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.901 -> 0.878 | Hurt |
| Add released-dataset information to the title and abstract (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.901 -> 0.915 | Helped |
| Add SOTA-claim information to the title and abstract (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.901 -> 0.881 | Hurt |
| Add all listed additional information (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.901 -> 0.917 | Helped |
| Add reference quality metric information (ablation) | Fine-tuned LLaMA-3 · NAID test set | NDCG: 0.901 -> 0.931 | Helped |
| Fine-tune the backbone with LoRA instead of training only the classification head (ablation) | LLaMA-3 · NAID test set | NDCG: 0.839 -> 0.901 | Helped |
Surprising findings
- Zero-shot ChatGPT prompting originally gave NDCG around 0.05 in the reproduction, rising to 0.597 only after adopting LLaMA-3's prompt.
- Adding external information such as dataset release or reference quality improves NDCG only in some cases, and open-access status lowers NDCG.
Limitations
- Dataset limited to about 12,000 entries because of Semantic Scholar API rate limits.
- Key phrase extraction depends on gpt-3.5-turbo-0125 and the TNCSI_SP target depends on Semantic Scholar citation counts, which are lower than Google Scholar counts.
- Predictions are probabilistic and the authors caution they should not replace peer review; manipulation of titles and abstracts is a risk.
Experimental setup
Datasets: NAID TKPD
Benchmarks: NAID test set
Metrics: MAE NDCG@20 NED
Baselines: MLP-based LSTM-based Model Ensemble ChatGPT-generated Phi-3 Falcon Qwen-2 Mistral
Builds on: TNCSI LoRA FWCI
Closest papers by meaning
- HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction (NeurIPS 2024 · similarity 0.82)
- SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding (ICLR 2025 · similarity 0.82)
- ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code (NeurIPS 2025 · similarity 0.81)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.80)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.80)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.