Research map / Large language models / AI and society
CHASM: Unveiling Covert Advertisements on Chinese Social Media
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CHASM, a manually annotated dataset of 4,992 RedNote posts for detecting covert advertisements, benchmarks 15 MLLMs zero-shot and in-context, and shows fine-tuning Qwen2.5-7B reaches 75.6% F1.
Problem
Covert advertisements disguised as regular social media posts deceive consumers, yet existing moderation benchmarks for LLMs/MLLMs do not cover them, and no labeled dataset exists for detecting them.
Approach
A new task definition and evidence-based guideline for covert ad detection, plus a curated multimodal (image, text, comments) dataset from RedNote that deliberately includes hard non-ad product-sharing posts, with dynamic quality-controlled annotation.
Key results
- GPT-4o achieves the best zero-shot F1 of 0.597 and DeepSeek-V3 the best in-context F1 of 0.592 on CHASM, judged not sufficiently reliable.
- Fine-tuned Qwen2.5-7B reaches F1 0.756 and AUC 0.852, surpassing zero-shot GPT-4o (F1 0.597).
- Removing images or comments significantly degrades the fine-tuned Qwen2.5-7B model's performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-7B on CHASM | Qwen2.5-7B · CHASM | F1: 0.421 -> 0.756 | Helped |
| Fine-tune InternVL on CHASM | InternVL · CHASM | F1: 0.403 -> 0.590 | Helped |
| Add detailed evaluation criteria to the prompt (ablation) | GPT-4o zero-shot · CHASM | F1: 0.596 -> 0.562 | Hurt |
| Add detailed evaluation criteria to the prompt (ablation) | DeepSeek-VL3 in-context learning · CHASM | F1: 0.592 -> 0.569 | Hurt |
| Remove images from the input (ablation) | Fine-tuned Qwen2.5-7B · CHASM | performance | Hurt |
| Remove comments from the input (ablation) | Fine-tuned Qwen2.5-7B · CHASM | performance | Hurt |
| Use in-context examples instead of zero-shot prompting | MLLMs on CHASM · CHASM | F1 | Mixed |
Surprising findings
- Reasoning MLLMs such as Gemini 2.5 Pro and Step-R1-V-Mini do not significantly outperform non-reasoning models on covert ad detection.
- More detailed guideline-laden prompts reduced performance compared with the normal prompt.
Limitations
- Limited to the RedNote platform and Chinese language.
- Dataset is relatively small due to limited human resources and lacks user behavior and creator profile data.
- Annotation is subjective with moderate initial inter-annotator agreement (Fleiss' kappa 0.65).
Experimental setup
Datasets: CHASM
Benchmarks: CHASM
Metrics: Precision Recall F1-Score AUC Accuracy
Baselines: GPT-4o DeepSeek-V3 Gemini 2.5 Pro Gemini 2.0 Qwen-Max InternVL2.5 LLaVA-NeXT TF-IDF + LR CLIP
Builds on: LoRA
Closest papers by meaning
- Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection (AAAI 2024 · similarity 0.83)
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.82)
- A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings (NeurIPS 2025 · similarity 0.81)
- Towards Comprehensive Detection of Chinese Harmful Memes (NeurIPS 2024 · similarity 0.81)
- ScamNet: Toward Explainable Large Language Model-Based Fraudulent Shopping Website Detection (AAAI 2025 · similarity 0.81)
- STLC-KG:A Social Text Steganalysis Method Combining Large-Scale Language Models and Common-Sense Knowledge Graphs (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.