Research map / Large language models / Alignment and preferences
Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stream Aligner, a small sentence-level corrector that iteratively rewrites the latest sentence of an upstream LLM's output and feeds the correction back into the prefix, improving helpfulness, harmlessness and math accuracy.
Problem
Existing alignment methods either need expensive training of the base model or, for inference-time add-on correctors like Aligner, rely heavily on the capability of the additional model and add latency because correction happens only once on the full answer.
Approach
Instead of correcting a full response once, a small model is fine-tuned on sentence-level preference pairs (original vs. corrected last sentence given prefix) and is run in a loop: the upstream model generates a sentence, the corrector rewrites it, and the corrected sentence is appended to the prefix for the next round. This elicits the upstream model's latent capability with a much smaller correc
Key results
- Llama2-70B-chat + Stream Aligner-2B reaches a maximum win-rate improvement of 76.1% in helpfulness and 36.0% in harmlessness (GPT-4 judged, direct pipeline).
- Stream Aligner-8B improves Llama3-70B-Instruct math accuracy by about 3.5% (reported in abstract).
- Stream Aligner achieves about 0.80x the per-token time of Aligner and first-token latency about 10x lower; SFT and DPO gave -0.5% and 0.3% math accuracy vs 5.8% for Stream Aligner on Llama3-8B-Instruct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace whole-answer correction with sentence-level streaming correction | Llama2-70B-chat with Stream Aligner-2B · Helpful QA | win rate: 21.3 -> 76.1 | Helped |
| Replace whole-answer correction with sentence-level streaming correction | Llama2-70B-chat with Stream Aligner-2B · Harmless QA | win rate: 7.2 -> 36.0 | Helped |
| Use Continue rather than Direct generation (ablation) | Stream Aligner-2B + Llama2-70B-chat · Helpful QA | win rate: Direct: 76.1; Continue: 54.3 | Mixed |
| Use Continue rather than Direct generation (ablation) | Stream Aligner-2B + Llama2-70B-chat · Harmless QA | win rate: Direct: 36.0; Continue: 50.0 | Mixed |
| Increase the number of generation-correction cycles (ablation) | Stream Aligner · Helpful and harmless QA; math | win rate / accuracy | Helped |
| Use Continue generation rather than the classic sentence-by-sentence correction pipeline (ablation) | Stream Aligner · Math | win rate | Helped |
| Use Stream Aligner-2B rather than Aligner-70B (ablation) | Llama3-70B-Instruct · Math | accuracy improvement | Helped |
| Increase the additional model size from Stream Aligner-2B to Stream Aligner-8B (ablation) | Llama3-8B-Instruct · Math | accuracy improvement: 6.1% -> 11.3% | Helped |
| Use SFT instead of Stream Aligner (ablation) | Llama3-8B-Instruct · Math | accuracy improvement: 5.8% -> -0.5% | Hurt |
| Use DPO instead of Stream Aligner (ablation) | Llama3-8B-Instruct · Math | accuracy improvement: 5.8% -> 0.3% | Hurt |
| Use Stream Aligner rather than Aligner with equal-sized correction models | Upstream model with correction models of equivalent size · Inference | per-token inference time: 0.80 times | Helped |
| Use Stream Aligner rather than Aligner | Inference pipeline · Inference | first-token latency: 10 times | Helped |
Surprising findings
- A 2B corrector applied to Llama2-70B-chat gives larger gains than the 7B Aligner, suggesting the upstream model's latent capability is under-used by single-shot correction.
- Stream Aligner-2B matches the performance of a 70B Aligner on math, despite being far smaller.
Limitations
- Still adds inference-time overhead from running the corrector each round.
- Requires relatively high-quality sentence-level training data and small correctors struggle with out-of-distribution inputs.
- Evaluation limited to helpful/harmless QA and math; judged mainly by GPT-4 with noted instability.
Experimental setup
Datasets: HH-RLHF PKU-SafeRLHF MATH BeaverTails
Benchmarks: HH-RLHF PKU-SafeRLHF MATH BeaverTails
Metrics: win rate (helpfulness) win rate (harmlessness) accuracy improvement (math) first-token latency per-token inference time
Baselines: Aligner Proxy-tuning SFT DPO Stream Aligner continue-generation pipeline
Builds on: Aligner speculative sampling proxy-tuning SFT
Closest papers by meaning
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.93)
- W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search (AAAI 2026 · similarity 0.89)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.89)
- LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations (NeurIPS 2026 · similarity 0.89)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.89)
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.