Research map / Large language models / Instruction and fine-tuning

What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

The paper analyzes why bidirectional fine-tuning of causal LLMs hurts generation, attributing it to increased subsequent attention dependence, and proposes UBMoE-LLM, a two-expert MoE that pairs the original unidirectional FFN with a contrastively fine-tuned bidirectional FFN.

Problem

Bidirectional fine-tuning of unidirectional LLMs improves embeddings but severely degrades generative performance; the cause of this degradation is unexplained.

Approach

Attention weights are used as dependence indicators (preceding/subsequent/self), showing bidirectional training raises subsequent dependence; FFN layers are least affected, so only the FFN is bidirectionally fine-tuned via contrastive learning and combined with the original FFN through a token-level gated MoE.

Key results

• On TruthfulQA, UBMoE-LLM improves over Qwen1.5-Chat at 0.5B (43.9 vs 43.0), 4B (47.6 vs 44.5) and 7B (55.4 vs 53.7), but drops on MMLU at 0.5B (28.1 vs 32.5) and 7B (59.6 vs 60.2).
• Fine-tuning only the FFN during bidirectional contrastive training gives STS average comparable to or better than full fine-tuning and yields lower subsequent dependence than ATT-only updates across scales (e.g., Qwen1.5-7B FFN STS avg 71.35 vs ALL 51.23 at 4-7B-level tables).
• UBMoE-LLM adds negligible compute: Qwen2.5-7B-Instruct 7,239.97 vs 7,240.18 GFLOPs.

Measured effects reported in the paper

Surprising findings

• Mixed causal+Prefix-I training slightly increased unidirectional generation performance on DOQA, suggesting some bidirectional benefit for long-text understanding.
• Bidirectional fine-tuning of the full model (ALL) gave the worst STS results at larger scales in the ablation, while FFN-only updates were strongest.

Limitations

• Generation degradation on some tasks (e.g., MMLU at 0.5B and 7B) is not eliminated, attributed to bidirectional experts lacking instruction alignment.
• Gains on TruthfulQA are modest and mixed across scales; many generation results are within a point of the base model.
• Evaluation is limited to a few benchmarks and the dependence metric's link to generation is correlational.

Experimental setup

Datasets: Li & Li 2024a contrastive data (480,862 examples) tulu-v2-sft-mixture (326,154 examples) MTEB STS tasks

Benchmarks: MMLU Winogrande TruthfulQA MTEB STS (10 tasks) Natural Instructions DOQA Dureader CMMLU Math PIQA

Metrics: accuracy mc2 Spearman correlation F1 Rouge-L BLEU average attention dependence (DP-main/before/after) GFLOPs

Baselines: Qwen1.5-Chat base models Llama3-8B MNTP-LLM LLM2Vec-style FFN+ATT update BatGPT

Builds on: LLM2Vec Prefix-LM attention SimCSE MTEB Mixtral-style MoE gating

Closest papers by meaning

• Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.86)
• The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.86)
• UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation (NeurIPS 2024 · similarity 0.86)
• KV Shifting Attention Enhances Language Modeling (ICML 2025 · similarity 0.85)
• Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting (NeurIPS 2025 · similarity 0.85)
• Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.85)