Research map / Adversarial robustness and security / Attacks on text and graphs

PhishAgent: A Robust Multimodal Agent for Phishing Webpage Detection

AAAI 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: LLM jailbreak and injection defense · 18 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PhishAgent, a multimodal LLM agent that combines online (Google search) and offline brand knowledge bases with text and image brand extractors and a retriever to detect phishing webpages.

Problem

Phishing webpage detection is hard because attackers evade existing methods and adversarial manipulations (prompt injection, brand spoofing, typosquatting) can fool single-model detectors.

Approach

Combine a multimodal retriever over a brand knowledge base with separate text (LLM) and image (MLLM) brand extractors, verified by domain and brand checks against online search and offline knowledge bases, plus a recheck step that re-derives the brand from the screenshot under adversarial conditions.

Key results

• On SG-SCAN-1k, PhishAgent with separate text and image brand extractors reaches 94.30% accuracy and 94.12 F1 at 2.54s per sample.
• Adding the Multimodal Retriever raises MBE target-brand recall from 66.28% (without MR) to 73.94% at k=5.
• Using one MLLM to process all modalities at once lowers PhishAgent accuracy to 90.9% and raises time to 4.90s.

Measured effects reported in the paper

Surprising findings

• Processing HTML, URL and screenshot in a single MLLM pass reduces both accuracy and speed compared with the separated design.
• Increasing k beyond about 10 hurts the brand extractor's recall even though retriever recall keeps rising.

Limitations

• Relies on commercial LLM/MLLM APIs (GPT-3.5-turbo-instruct, GPT-4o) and Google search, incurring cost and dependence on external services.
• Search-engine query costs apply beyond a daily free allowance.

Experimental setup

Datasets: TR-OP TR-AP SG-SCAN-1k SG-SCAN_unl OpenPhish_train

Benchmarks: TR-OP TR-AP SG-SCAN-1k

Metrics: accuracy F1 precision recall running time

Baselines: ChatPhishDetector GEPAgent KPD Phishpedia

Builds on: Phishpedia ChatPhishDetector GEPAgent KPD

Closest papers by meaning

• 2-in-1 Phishing Detection via Large LM Distillation and Small LM Perturbation (Student Abstract) (AAAI 2025 · similarity 0.82)
• ScamNet: Toward Explainable Large Language Model-Based Fraudulent Shopping Website Detection (AAAI 2025 · similarity 0.82)
• Teach LLMs to Phish: Stealing Private Information from Language Models (ICLR 2024 · similarity 0.82)
• DAMMFND: Domain-Aware Multimodal Multi-view Fake News Detection (AAAI 2025 · similarity 0.80)
• SafeArena: Evaluating the Safety of Autonomous Web Agents (ICML 2025 · similarity 0.80)
• MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.80)