Research map / Large language models / Language, safety and interpretability

From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds FineHarm (29K prompt-response pairs with POS-based token-level harm labels) and trains a Qwen2.5-based Streaming Content Monitor with holistic/token dual supervision and a logic-consistency loss for early stopping of harmful LLM outputs.

Problem

Existing content moderators judge harmfulness only on complete LLM outputs, causing latency; partial detection moderators trained on full responses suffer a training-inference gap, and fine-grained token-level harmful annotations are lacking.

Approach

A streaming token-level harm monitor (SCM) trained with hierarchical consistency-aware learning, where a holistic response-level scorer on the last token is tied to token-level scores via a propositional logic loss; token labels come from a sentence-then-POS heuristic annotation, and Delay-k rule decides early termination.

Key results

• SCM-7B achieves 97.45 macro F1 on FineHarm, above full-detection Qwen2.5-7B (92.75), by seeing on average the first 18% of tokens.
• Over 80% of harmful responses are detected within the first 30% of tokens.
• Token-level DPO using SCM annotations gives higher harmlessness than DPO on several PKU-SafeRLHF categories (e.g., Emotional harm 7.20 vs 2.60) while keeping comparable MT-Bench helpfulness.

Measured effects reported in the paper

Surprising findings

• Full-detection ModernBERT and Qwen2.5 drop substantially when applied as partial detectors without partial-aware training (e.g., Qwen2.5-7B macro F1 falls from 92.75 to 78.60).
• Sample of token-level DPO yields lower MT-Bench helpfulness than DPO (5.42 vs 5.61) despite higher harmlessness in some categories.

Limitations

• Token-level labels are heuristic (sentence-level LLM labels plus POS filtering) and may miss nuanced harmfulness.
• Performance depends on coverage and representativeness of the training data; OOD performance on ToxicChat and ToxiGen is moderate.
• The safety alignment application is preliminary and only tested on one 8B model.

Experimental setup

Datasets: FineHarm WildGuard WildJailbreak PKU-SafeRLHF MT-Bench ToxicChat ToxiGen

Benchmarks: FineHarm ToxicChat ToxiGen MT-Bench PKU-SafeRLHF

Metrics: macro F1 precision recall F1 helpfulness score harmlessness score latency

Baselines: HateBERT ToxDectRoBERTa Google Perspective OpenAI Moderation LlamaGuard-3-8B ModernBERT Qwen2.5 (partial detection) DPO

Builds on: Qwen2.5 ModernBERT DPO

Closest papers by meaning

• Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.88)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.88)
• BingoGuard: LLM Content Moderation Tools with Risk Levels (ICLR 2025 · similarity 0.88)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.87)
• Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.87)