Research map / Large language models / Language, safety and interpretability
When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FLUB, an 834-sample Chinese benchmark of tricky "cunning" texts from the Ruozhiba forum, testing LLMs on answer selection, cunning type classification, and fallacy explanation.
Problem
LLMs are trained on cleaned, correct corpora and may fail to understand fallacious, misleading, or humorous texts that humans find easy to parse.
Approach
Collects real cunning posts from the Chinese Ruozhiba forum, annotates fine-grained fallacy types, correct explanations, and three confusing wrong candidates, and evaluates LLMs on three tasks of increasing difficulty.
Key results
- GPT-4-Turbo achieves the best overall performance (22.00 geometric-mean overall with CoT) and 82.73% accuracy on answer selection with CoT.
- All models score below 15.0 F-1 on cunning type classification; human annotators reach 63.69 F-1 and 93.35% selection accuracy.
- GPT-4 automatic scores correlate with human ratings at Spearman 0.69 overall on 50 sampled explanations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Chain-of-Thought prompting (ablation) | GPT-4-Turbo · FLUB | Accuracy: 79.38 -> 82.73 | Helped |
| Add Chain-of-Thought prompting (ablation) | ERNIE-Bot-4.0 · FLUB | Accuracy: 75.66 -> 71.34 | Hurt |
| Add Chain-of-Thought prompting (ablation) | GPT-4-Turbo · FLUB | F-1 Score: 12.31 -> 13.97 | Helped |
| Add Chain-of-Thought prompting (ablation) | GPT-4-Turbo · FLUB | GPT-4 Score: 8.95 -> 9.21 | Helped |
| Add Chain-of-Thought prompting (ablation) | ChatGLM3-6B · FLUB | GPT-4 Score: 4.98 -> 4.82 | Hurt |
| Add Chain-of-Thought prompting (ablation) | Qwen-72B-Chat · FLUB | Overall Performance: 15.89 -> 18.13 | Helped |
| Add Chain-of-Thought prompting (ablation) | Yi-6B-Chat · FLUB | Overall Performance: 11.81 -> 11.58 | Hurt |
| Increase in-context demonstrations from zero-shot to more examples (ablation) | Five high-performing LLMs · FLUB | Performance | Helped |
| Add one in-context demonstration (ablation) | Five high-performing LLMs · FLUB | Performance | Mixed |
| Increase Qwen model parameter size | Qwen models · FLUB | F-1 Score: 10.34 -> 9.32 | Hurt |
| Increase Yi model parameter size | Yi models · FLUB | F-1 Score: 8.87 -> 4.74 | Hurt |
| Use larger model scale | LLMs evaluated on FLUB · FLUB | Overall Performance | Helped |
Surprising findings
- Chain-of-Thought does not give stable gains and often hurts answer selection and explanation tasks.
- Larger Qwen and Yi models perform worse on cunning type classification as size increases.
Limitations
- Benchmark consists only of Chinese data and relies on Chinese cultural and language knowledge.
- Small dataset size (834 samples) with highly imbalanced cunning type distribution.
- Automatic explanation scoring relies on GPT-4 judge with moderate human correlation (0.42 to 0.74 per model).
Experimental setup
Datasets: Ruozhiba forum posts (Baidu Tieba)
Benchmarks: FLUB
Metrics: accuracy F-1 score GPT-4 score Spearman correlation
Baselines: ERNIE-Bot-3.5 ERNIE-Bot-3.5-Turbo ERNIE-Bot-4.0 GPT-3.5-Turbo GPT-4-Turbo ChatGLM3-6B Qwen-7B-Chat Qwen-14B-Chat Qwen-72B-Chat Yi-6B-Chat Yi-34B-Chat Baichuan2-7B-Chat
Builds on: MT-Bench TruthfulQA
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.89)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.