Research map / Vision-language and multimodal / Visual question answering
Fool Your (Vision and) Language Model with Embarrassingly Simple Permutations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs and VLLMs are highly sensitive to permutations of multiple-choice answer options, with brute-force adversarial permutation dropping accuracy often below chance, and that position-bias calibration, majority voting, PriDe, and permutation-augmented fine-tuning only partially help.
Problem
Multiple-choice question answering (MCQA) evaluations of LLMs and vision-language LLMs may overstate robustness, since models may change answers when only the order of answer options is permuted.
Approach
Brute-force search over all answer-option permutations to find one that makes the model err (an adversarial permutation attack), showing the vulnerability goes beyond position bias and is not fixed by calibration, majority voting, or PriDe.
Key results
- Llama2-13B's MMLU accuracy falls from 52.22% to 18.33% under the permutation attack, below the 25% chance level.
- Llama2-7B's BoolQ accuracy drops from 61.79% to 8.23% under the attack, well below the 50% chance level.
- Majority vote and M-Confidence recover only part of the gap, and contextual calibration fails to help (e.g., Llama2-7B MMLU 6.17 after attack vs. 5.24 with calibration).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Randomly permute answer-option positions | LLMs on MCQA benchmarks · MMLU | accuracy: 52.22/18.33 (33.89 $\downarrow$) | Hurt |
| Randomly permute answer-option positions | Llama2-7B · BoolQ | accuracy: 61.79/\phantom{0}8.23 (53.56 $\downarrow$) | Hurt |
| Randomly permute answer-option positions | VLLMs on MCQA benchmarks · ScienceQA | accuracy: 71.60/52.55 (19.05 $\downarrow$) | Hurt |
| Reduce answer set to 2 choices, retaining the true answer, and permute | Llama2-7B · MMLU | accuracy: 63.33/20.26 (43.07$\downarrow$) | Hurt |
| Increase model size from 7B to 13B | InstructBLIP · ScienceQA | accuracy: 59.46/33.31 -> 64.15/41.84 | Helped |
| Use majority voting across answer-option permutations | Llama2-7B · MMLU | accuracy: 6.17 -> 33.64; original 40.91 | Mixed |
| Apply contextual calibration | Llama2-7B under permutation attack · MMLU | accuracy: 6.17 -> 5.24 | No clear effect |
| Select the maximum-confidence prediction across permutations | Llama2-7B · MMLU | accuracy: 6.17 -> 22.62; original 40.91 | Helped |
| Apply PriDe debiasing | Llama2-7B under permutation attack · MMLU | accuracy: 42.60/\phantom{0}6.43 (36.17$\downarrow$) | No clear effect |
| Fine-tune on the original training data | Llama2-7B · ARC-Challenge | accuracy: 47.04/ 7.98 -> 51.42/15.02 | Helped |
| Add option permutations to fine-tuning data | Llama2-7B · ARC-Challenge | accuracy: 47.04/ 7.98 -> 67.64/47.73 | Helped |
| Use circular evaluation instead of adversarial permutation | LLMs · MMLU | accuracy: 27.26 vs 6.17 | Mixed |
Surprising findings
- Even with only two answer options left, most models fall below chance under permutation, e.g., Llama2-7B drops to 20.26% on 2-choice MMLU versus a 50% chance level.
- Fine-tuning with option permutations only partially alleviates the vulnerability, and regular fine-tuning does not fix it.
Limitations
- Attack is brute-force over k! permutations, which would scale poorly to large k.
- No proposed fix; mitigations evaluated are post-hoc and fail to fully restore accuracy.
Experimental setup
Datasets: MMLU ARC-Challenge BoolQ SocialIQA MedMCQA ScienceQA A-OKVQA MMBench SEED-Bench
Benchmarks: MMLU ARC-Challenge BoolQ SocialIQA MedMCQA ScienceQA A-OKVQA MMBench SEED-Bench
Metrics: accuracy accuracy drop relative to original chance-level comparison
Baselines: contextual calibration majority vote PriDe CircularEval symbol attack
Builds on: contextual calibration PriDe LoRA circular evaluation
Closest papers by meaning
- Set-LLM: A Permutation-Invariant LLM (NeurIPS 2025 · similarity 0.88)
- Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (CVPR 2024 · similarity 0.87)
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions (ICLR 2025 · similarity 0.86)
- PEARL: Towards Permutation-Resilient LLMs (ICLR 2025 · similarity 0.86)
- Large Language Models Are Not Robust Multiple Choice Selectors (ICLR 2024 · similarity 0.86)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.