Research map / Vision-language and multimodal / Multimodal LLMs
FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexAC, a training-free method that steers middle-layer hidden states of MLLMs with hallucination-derived associative vectors (plus task-specific GPT-4o-derived vectors and adaptive calibration) to bidirectionally control faithfulness versus creativity.
Problem
MLLMs face a trade-off between faithfulness (suppressing hallucination) and creativity (associative reasoning), and existing methods offer no flexible, task-dependent control over associative strength.
Approach
Treat hallucinated responses as associative signals, extract middle-layer difference vectors between hallucinated and grounded representations, and inject them at inference with a sigmoid-based intensity calibration and norm preservation, plus task-specific vectors from a few high-association samples to control both suppression and enhancement.
Key results
- On Qwen-VL, FlexAC reduces CHAIR_S from 40.6 to 19.2 and CHAIR_I from 12.5 to 5.4 in faithfulness mode.
- On Creation-MMBench with Qwen-VL, FlexAC achieves an overall Reward of 10.92 versus -3.86 for VCD and -1.63 for VAF.
- On VDAT with LLaVA-1.5, FlexAC raises the score to 88.49 versus 86.89 for the regular model, while Ha-DPO drops it to 85.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| suppress associative behavior with FlexAC | Qwen-VL · CHAIR | CHAIR_S: 40.6 -> 19.2 | Helped |
| suppress associative behavior with FlexAC | LLaVA-1.5 · CHAIR | CHAIR_S: 50.8 -> 36.6 | Helped |
| suppress associative behavior with FlexAC | DeepSeek-VL2 · CHAIR | CHAIR_S: 32.6 -> 28.6 | Helped |
| enhance associative behavior with FlexAC | LLaVA-1.5 · VDAT | VDAT score: 86.89 -> 88.49 | Helped |
| enhance associative behavior with FlexAC | Qwen-VL · Creation-MMBench Overall | Reward: 0.00 -> 10.92 | Helped |
| remove Instance Selection and Steering Intensity Calibration (ablation) | FlexAC faithfulness variant on Qwen-VL · CHAIR_S | CHAIR_S: 19.2 -> 30.4 | Hurt |
| remove Instance Selection and Steering Intensity Calibration (ablation) | FlexAC creativity variant on Qwen-VL · VDAT | VDAT score: 86.58 -> 85.05 | Hurt |
| replace associative steering with random vectors (ablation) | FlexAC on LLaVA-1.5 · Creation-MMBench Overall | Reward: 4.39 -> -60.49 | Hurt |
| replace associative steering with random vectors (ablation) | FlexAC on DeepSeek-VL2 · Creation-MMBench Overall | Reward: 6.73 -> -77.47 | Hurt |
| control middle layers rather than shallow or deep layers (ablation) | FlexAC · CHAIR and VDAT | CHAIR and VDAT scores | Helped |
| vary the number of selected instances; performance stabilizes near K = 50 (ablation) | FlexAC on Qwen-VL · CHAIR and VDAT | CHAIR and VDAT scores: K = 50 | Mixed |
| add Directional Integration (ablation) | FlexAC creativity variant · VDAT | VDAT score | Helped |
Surprising findings
- Hallucination-suppressing Ha-DPO reduces VDAT creativity score below the regular model, showing faithfulness and creativity trade off.
- Random-vector injection causes large drops on Creation-MMBench, indicating gains are not from perturbation noise.
Limitations
- Requires white-box access to hidden states, so not applicable to black-box models such as ChatGPT.
- Faithfulness mode reduces recall on CHAIR and POPE for some models (e.g., Qwen-VL recall 62.5 vs 71.7).
- Layer selection is model-specific and chosen by empirical sweeps.
Experimental setup
Datasets: COCO2014 MSCOCO Creation-MMBench
Benchmarks: CHAIR POPE VDAT Creation-MMBench MME MMMU MMStar
Metrics: CHAIR_S CHAIR_I F1 accuracy VDAT score VFS Reward
Baselines: Regular VCD VAF Ha-DPO
Builds on: Visual Contrastive Decoding HA-DPO representation engineering
Closest papers by meaning
- MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models (CVPR 2026 · similarity 0.88)
- The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs (ICLR 2025 · similarity 0.88)
- Reallocating Attention Across Layers to Reduce Multimodal Hallucination (CVPR 2026 · similarity 0.88)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.