Research map / Large language models / Language, safety and interpretability
Combating Phone Scams with LLM-based Detection: Where Do We Stand? (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes LLMs (GPT-4, GPT-4o, GLM-4, Doubao-Pro-32k, ERNIE-3.5-8k) for detecting phone scam dialogues, showing that existing scam datasets are biased and that LLM detectors trade off recall and suffer from inconsistency.
Problem
Phone scam detection from call transcripts: existing public scam dialogue datasets may be biased, classic classifiers can be evaded by avoiding keywords, and LLM-based detectors face low recall and hallucination-like inconsistency.
Approach
Shows that TF-IDF-based classifiers reach near-perfect scores on public scam datasets only because of keyword artifacts (performance collapses on LLM-generated adversarial conversations), then evaluates zero-shot LLM detectors on real and synthetic calls and measures their answer consistency with a discrete semantic entropy-based reliability score.
Key results
- Random Forest on public scam datasets (SC, SD, MASC) reaches accuracy/F1 of about 0.99-1.00, but drops to accuracy 0.53 and F1 0.26 on the adversarially generated Our-Synt set.
- LLM detectors achieve precision above 0.98 on Our-Real but recall as low as 0.72; on synthetic data precision exceeds 0.95 and recall as low as 0.86.
- GPT-4o reaches a perfect reliability score of 1 on Our-Real, while GLM-4 on Our-Synt scores only 0.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the top 50 terms from real conversations in newly generated fraudulent and genuine samples | TF-IDF Random Forest classifiers · Our-Synt | accuracy: 1.00 -> 0.53 | Hurt |
| Use LLM-based detection instead of traditional classifiers | Phone scam detection · Our-Real | detection performance | Helped |
| Use LLM-based detection | Phone scam detection · Our-Real | recall: as low as 0.72 | Hurt |
| Use LLM-based detection | Phone scam detection · Our-Synt | recall: as low as 0.86 | Hurt |
| Repeat model predictions across runs | LLM-based scam detection · Phone scam conversations | reliability | Hurt |
Surprising findings
- Random Forest, SVM and KNN classifiers near-perfectly separate public scam datasets, yet collapse on keyword-avoiding synthetic conversations, indicating keyword shortcut learning.
Limitations
- Public scam datasets are synthesized and biased toward keywords
- LLM recall is relatively low (as low as 0.72), leaving many scams undetected
- Inconsistent responses across repeated runs indicate hallucination risk
Experimental setup
Datasets: SC SD MASC Our-Real Our-Synt
Benchmarks: SC SD MASC Our-Real Our-Synt
Metrics: accuracy precision recall F1-score reliability score
Baselines: Support Vector Machines Random Forest k-Nearest Neighbors
Builds on: discrete semantic entropy
Closest papers by meaning
- ScamNet: Toward Explainable Large Language Model-Based Fraudulent Shopping Website Detection (AAAI 2025 · similarity 0.88)
- Can LLM-Generated Misinformation Be Detected? (ICLR 2024 · similarity 0.85)
- HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice (NeurIPS 2026 · similarity 0.85)
- Can We Infer Confidential Properties of Training Data from LLMs? (NeurIPS 2025 · similarity 0.84)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.84)
- Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.