Research map / Large language models / Instruction and fine-tuning
Walking the Tightrope: Autonomous Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Counterfactual Preference Optimization (CPO), a DPO-style RFT method that uses concept-graph-guided LLM counterfactual CoTs as negatives to disentangle detrimental concept drift in chest X-ray diagnosis with Qwen2.5-VL-7B, and releases the 320K-trajectory CXR-CounterFact dataset.
Problem
Detrimental concept drift in chain-of-thought reasoning during non-stationary reinforcement fine-tuning of MLLMs, which biases final diagnostic predictions in chest radiograph analysis.
Approach
Builds a hierarchical disease-attribute concept graph with association, irrelevance, and exclusion relations, uses it to prompt an LLM expert (Med-PaLM) to generate counterfactual diagnostic reasoning as negative samples, and optimizes a DPO-style counterfactual preference loss contrasting radiologist reports against these counterfactuals.
Key results
- CPO reaches 81.8% average top-1 accuracy on MS-CXR-T, versus 73.8% for SFT and 69.2% for the second-best baseline CoCa-CXR.
- CPO reports BLEU-4 0.155 and METEOR 0.236 on MIMIC-CXR report generation, the best among compared methods.
- In zero-shot classification CPO achieves the highest AUC on all six benchmarks, e.g. 84.4 on Open-I and 92.5 on CheXpert versus CARZero's 83.8 and 92.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CPO instead of SFT | Qwen2.5-VL (7B) · MS-CXR-T | Top-1 accuracy (Avg.): 73.8 -> 81.8 | Helped |
| Apply CPO instead of SFT | Qwen2.5-VL (7B) · MS-CXR-T, Con. | Top-1 accuracy: 54.9 -> 77.7 | Helped |
| Apply CPO instead of SFT | Qwen2.5-VL (7B) · MS-CXR-T, Pna. | Top-1 accuracy: 70.0 -> 87.4 | Helped |
| Apply CPO instead of SFT | Qwen2.5-VL (7B) · MS-CXR-T, Pnx. | Top-1 accuracy: 95.9 -> 95.8 | Hurt |
| Apply CPO instead of SFT | Qwen2.5-VL (7B) · MS-CXR-T, Ede. | Top-1 accuracy: 77.1 -> 75.3 | Hurt |
| Use CPO instead of the strongest baseline per metric | Chest X-ray report generation · MIMIC-CXR | BLEU-4: 0.133 -> 0.155 | Helped |
| Use CPO instead of the strongest baseline per metric | Chest X-ray report generation · MIMIC-CXR | ROUGE-L: 0.291 -> 0.321 | Helped |
| Use CPO instead of the strongest baseline per metric | Chest X-ray report generation · MIMIC-CXR | METEOR: 0.175 -> 0.236 | Helped |
| Use CPO instead of the strongest baseline per metric | Chest X-ray report generation · MIMIC-CXR | CIDEr: 0.362 -> 0.375 | Helped |
| Use CPO instead of CARZero | Zero-shot disease classification · Open-I, PadChest, PadChest20, ChestXray14, ChestXpert, ChestXDet10 | AUC: 83.8 -> 84.4; 81.0 -> 82.0; 83.7 -> 85.1; 81.1 -> 81.7; 92.3 -> 92.5; 79.6 -> 80.1 | Helped |
| Add CoT to RFT without CPO (ablation) | Qwen2.5-VL (7B) · MS-CXR-T | Top-1 accuracy (Avg.): 73.8 -> 74.9 | Helped |
| Add CPO to RFT without CoT (ablation) | Qwen2.5-VL (7B) · MS-CXR-T | Top-1 accuracy (Avg.): 73.8 -> 78.3 | Helped |
Surprising findings
- Ablation suggests CPO (counterfactual negatives without CoT) gives a larger gain (+4.5 avg) than CoT-based DPO (+1.1 avg) in RFT.
- SFT alone is weak on consolidation and pneumonia (54.9 and 70.0), which the authors attribute to concept drift in CoT.
Limitations
- Counterfactual trajectories are generated by Med-PaLM and not validated by clinicians beyond the stated plausibility claim.
- Authors note the gap to TempA-VLP on edema is attributed to extra Chest ImaGenome annotations, making comparison unequal.
- Theoretical link between concept drift and RFT is largely informal; no formal drift guarantees are shown.
Experimental setup
Datasets: MIMIC-CXR CXR-CounterFact (CCF) MS-CXR-T Open-I PadChest PadChest20 ChestXray14 CheXpert ChestXDet10
Benchmarks: MS-CXR-T MIMIC-CXR report generation Open-I PadChest PadChest20 ChestXray14 CheXpert ChestXDet10
Metrics: top-1 accuracy BLEU-1/2/3/4 ROUGE-L METEOR CIDEr AUC
Baselines: CTrans CheXRelNet BioViL BioViL-T Med-ST TempA-VLP CoCa-CXR R2Gen METransformer R2GenGPT MambaXray CARZero
Builds on: DPO Qwen2.5-VL Visual-RFT GRPO Med-PaLM
Closest papers by meaning
- Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments (ICML 2026 · similarity 0.90)
- RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models (ICLR 2026 · similarity 0.86)
- On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.86)
- Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.86)
- BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.86)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.