Research map / Continual learning / Continual fine-tuning of LLMs
DiNADO: Norm-Disentangled Neurally-Decomposed Oracles for Controlling Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves NADO controllable generation by norm-disentangling the step-wise oracle ratio (DiNADO variants), adding likelihood-reweighted importance sampling, and merging with LoRA finetuning of the base model.
Problem
Vanilla NADO for controllable generation suffers from non-unique optima during warmup SFT, gradient variance for low-probability control signals, limited capacity of its small adapter transformer, and poor sample efficiency.
Approach
Parameterize the NADO oracle ratio as a normalized per-step distribution r times a step-wise scalar beta (removing the scale ambiguity and the Bellman regularizer in the Hard variant), reweight training samples by base-model likelihood, and combine with LoRA by treating the LoRA-modified model's ratio to the base model as the NADO output (DiNADO-Merge).
Key results
- DiNADO-Merge reaches 34.2 BLEU (test, GPT-synthesized references) and 98.9% coverage on CommonGen, matching GeLaTo's 34.1 BLEU and 100% coverage on the original reporting and exceeding NADO's 30.7 BLEU.
- On FormalMT, DiNADO-Merge with full finetuning reaches 21.58 formal BLEU and 0.60 formality versus 20.76 and 0.53 for NADO.
- With likelihood-based reweighting, DiNADO-Soft reaches 28.6 BLEU at N=4 samples per input, versus 25.1 for NADO at N=4 (NADO needs N=32 to reach 30.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiNADO-Merge with full-parameter finetuning instead of NADO | GPT-2 Large · FormalMT | Formal BLEU: 20.76 -> 21.58 | Helped |
| Use DiNADO-Merge with LoRA instead of NADO | GPT-2 Large · FormalMT | Formal BLEU: 20.76 -> 21.37 | Helped |
| Use DiNADO-Merge instead of GeLaTo | GPT-2 Large · CommonGen test | BLEU: 34.1 -> 34.2* | Helped |
| Remove reranking (ablation) | DiNADO-Merge · CommonGen test | BLEU: 34.2* -> 32.9 | Hurt |
| Replace soft consistency regularization with exact forward consistency (ablation) | DiNADO · CommonGen test | BLEU: 31.4* -> 29.0* | Hurt |
| Normalize the parameterization during warm-up instead of using unnormalized NADO (ablation) | GPT-2 Large adaptation layers · CommonGen dev | BLEU: 13.98 -> 16.36 | Helped |
| Use a smaller independent model rather than adaptation layers (ablation) | Normalized DiNADO · CommonGen dev | BLEU: 16.36 -> 17.15 | Helped |
| Use normalized DiNADO-Merge rather than normalized adaptation layers (ablation) | GPT-2 Large · CommonGen dev after SFT | BLEU: 16.36 -> 28.91 | Helped |
| Add likelihood-based reweighting (ablation) | DiNADO-Soft-GPT-2 Base · CommonGen test, N=4 | BLEU: 25.1* -> 29.1* | Helped |
| Add likelihood-based reweighting (ablation) | DiNADO-Soft-GPT-2 Base · CommonGen test, N=2 | BLEU: 23.7* -> 24.7* | Helped |
| Increase samples per input from N=2 to N=64 (ablation) | NADO-GPT-2 Base · CommonGen test | BLEU: 22.6* -> 30.6* | Helped |
| Increase samples per input from N=2 to N=64 (ablation) | DiNADO-Soft-GPT-2 Base with likelihood-based reweighting · CommonGen test | BLEU: 24.7* -> 31.4* | Helped |
Surprising findings
- Adaptation-layer NADO and DiNADO-Hard underperform the plain GPT-2 Base or Flan-T5 baselines in some warmup settings, with the composed distribution sometimes worse than direct sampling from the base model.
- DiNADO-Merge with LoRA-style finetuning matches or exceeds full-parameter finetuning after warmup, which the authors attribute to identical behavior with vanilla LoRA during SFT.
Limitations
- Main results rely on GPT-synthesized pseudo test references for CommonGen because the official test server is closed.
- Evaluation is limited to two tasks (formality MT and CommonGen) with small base models (GPT-2, T5, Flan-T5).
- DiNADO-Hard performs worse than DiNADO-Soft in practice, so the theoretically cleaner formulation does not win empirically.
Experimental setup
Datasets: CommonGen FormalMT (Fisher and CALLHOME Spanish-English)
Benchmarks: CommonGen FormalMT
Metrics: BLEU coverage formal BLEU formality
Baselines: NADO GeLaTo FUDGE NeuroLogic A*esque Direct Finetune
Builds on: NADO GeLaTo FUDGE LoRA
Closest papers by meaning
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.86)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.86)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.86)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.86)
- Adaptable Logical Control for Large Language Models (NeurIPS 2024 · similarity 0.86)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.