Research map / Large language models / Language, safety and interpretability

ReFF: Reinforcing Format Faithfulness in Language Models Across Varied Tasks

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FormatBench, a multi-task format-faithfulness benchmark with per-task format checkers, and ReFF, which uses format-checker rewards in PPO-style RL (LoRA on LLaMA3-8B), with or without labeled data, to raise format faithfulness while keeping general quality.

Problem

LLMs often fail to follow prescribed output formats across varied tasks, and existing format benchmarks cover narrow task sets while existing adaptation methods ignore that format compliance can be checked programmatically.

Approach

Treat format checker pass/fail as a reward in an RLHF-style PPO loop (with KL penalty) so the LLM is directly optimized for format faithfulness, optionally combined with supervised finetuning to recover general quality.

Key results

• ReFF-tst raises LLaMA3-8B caption segmentation format faithfulness from 21.6% to 95.0% (ReFF-tst-CapSeg) with F1 comparable (47.3 to 46.4).
• ReFF-trn-ft reaches 75.5% format faithfulness on CapSeg with general quality 61.6 F1, versus 47.3 for the original LLaMA3.
• Original LLaMA3-8B format faithfulness averages 54.8%, and GPT-3.5 averages 63.8% on FormatBench, showing the benchmark is challenging.

Measured effects reported in the paper

Surprising findings

• Smaller Falcon-inst achieves 54.5% format faithfulness on XDL versus 5.5% for GPT-3.5.
• RL-adapted model on XDL produced syntactically correct but irrelevant code, an instance of mode collapse that boosts format faithfulness while hurting quality.

Limitations

• Test-only setting applies only to offline batch scenarios where test queries are available beforehand.
• RL can induce mode collapse, yielding format-compliant but irrelevant outputs (e.g., XDL).
• General quality is not evaluated for XDL due to lack of references and expert need.

Experimental setup

Datasets: TREC question classification SQuAD CoNLL-2003 Penn Treebank MuST-Cinema WMT 2023 Terminology Shared Task Poem Hunter acrostic poems Kaggle Poems Dataset First TextWorld Problems XDL generation data

Benchmarks: FormatBench

Metrics: format faithfulness rate general quality (accuracy, F1, BLEU-4, GPT-4 score, game sco

Baselines: GPT-3.5 IFEval FoFo Struc-Bench Chem-* Self-refine-style refinement Finetuning on LLaMA3

Builds on: PPO RLHF CTG

Closest papers by meaning

• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.90)
• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.88)
• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.88)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.87)
• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.87)
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.87)