Research map / Multi-agent systems and games / LLM agents

BotSim: LLM-Powered Malicious Social Botnet Simulation

AAAI 2025 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BotSim, an LLM-driven social botnet simulation framework on Reddit, and releases the BotSim-24 dataset of human and LLM-driven bot accounts, showing that bot detectors effective on traditional datasets perform worse on it.

Problem

Existing bot detection datasets and methods target traditional bots; LLM-driven bots with human-like profiles, text and interactions evade detection, and manual annotation is unreliable and ethically constrained for collecting real bot data.

Approach

An LLM-agent simulation framework (environment perception, action list, agent decision center with memory and background knowledge) that generates disguised LLM bots interacting with real Reddit human accounts, yielding a labeled bot-detection dataset with bot-human interaction graphs.

Key results

• On BotSim-24, BotRGCN reaches 89.9% accuracy and 86.7% F1, the best among 11 baselines, while Wei et al.'s text method drops to 50.8% accuracy.
• Text-based LLM detectors on BotSim-24 perform poorly: GPT-4-turbo 5-shot reaches 70.76% accuracy and 63.58% F1, and Llama3-8B zero-shot 54.82% accuracy.
• Edge perturbation shows GNN detection performance initially declines and then recovers as the share of reversed human-to-bot edges grows.

Measured effects reported in the paper

Surprising findings

• Adding Roberta text features to metadata (Roberta+NN) lowers performance on BotSim-24 (67.6% accuracy vs 75.7% for RF on metadata), indicating bot text disguise works.
• Graph methods do better on BotSim-24 partly because the dataset lacks human-to-bot edges, a structural artifact of the simulation.

Limitations

• Dataset is small (1,000 bots) due to LLM API cost constraints.
• Human data is pre-collected, so the simulation lacks real human-to-bot interactions, creating a structural artifact that favors GNNs.
• Only Reddit and one goal task family (disinformation on news events) are covered; human annotator study uses only 200 samples from three graduate students.

Experimental setup

Datasets: BotSim-24 Cresci-15 Cresci-17 TwiBot-20 TwiBot-22 MGTAB-22

Benchmarks: BotSim-24 Cresci-15 Cresci-17 TwiBot-20 TwiBot-22 MGTAB-22

Metrics: accuracy F1-score

Baselines: Adaboost decision tree random forest SVM Wei et al. text RNN Roberta+NN GCN GAT S-HGN BotRGCN RGT

Builds on: Reddit PRAW data collection RGT BotRGCN S-HGN Chain-of-Thought

Closest papers by meaning

• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards (ICML 2025 · similarity 0.83)
• Adversarial Socialbots Modeling Based on Structural Information Principles (AAAI 2024 · similarity 0.83)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.82)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.82)
• UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning (ICML 2025 · similarity 0.82)
• Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.82)