Research map / Large language models / Language, safety and interpretability

Time-Reversal Provides Unsupervised Feedback to LLMs

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Time Reversed Language Models (TRLMs) that score and generate queries conditioned on responses, via prompted forward models or reverse-token pre-trained models, and applies them to best-of-N reranking, citation attribution, retrieval, and jailbreak input-filter augmentation.

Problem

Forward LLMs score responses given queries, but this direction may give weak unsupervised feedback; the paper asks whether reverse (response-to-query) scoring and generation can provide complementary feedback without extra supervision.

Approach

Train or prompt language models to predict in the reverse token order (TRLM-Ba) or prompt forward models to predict queries from responses (TRLM-Fo), and use the reverse log-likelihood of the query given the response as a reranking score and the reverse generator to project candidate jailbreak responses back to query space for filtering.

Key results

• TRLM-Ba reranking of 16 Gemini-Pro-1.0 responses reaches 32.44 LC win rate on AlpacaEval, about 5 points above the self-scoring baseline (27.05).
• On NF-Corpus, TRLM-Fo reaches NDCG@10 of 48.08 versus 3.89 for the forward baseline.
• TRLM-Ba (PT) lowers FNR on the human-annotated set from 0.00 for TRLM-Fo to 18.18 at threshold 2, and reduces FNR on JBB by roughly 35% over the gpt-3.5 input filter alone.

Measured effects reported in the paper

Limitations

• Theoretical results rely on a stylized bipartite graph and hallucination model that may not reflect real language.
• Evaluation focuses on short-query, long-answer tasks; benefits on other tasks are unexplored.
• Jailbreak evaluation uses small datasets (about 68 JBB questions, about 100 HA questions) and GPT-4 as judge.

Experimental setup

Datasets: FLAN AlpacaEval (805 AlpacaFarm questions) CNN Daily Mail MS-Marco NF-Corpus JailbreakBench Alpaca Eval2 (E set)

Benchmarks: AlpacaEval CNN Daily Mail citation attribution MS-Marco NF-Corpus JailbreakBench MTEB retrieval

Metrics: length-controlled win rate win rate Gecko cosine similarity TF-IDF ROUGE precision recall NDCG@10 FNR FPR

Baselines: Self scoring (log-perplexity) Forward Baseline Backward Baseline One Generation gpt-3.5-turbo-1106 input filter perplexity thresholding defense SmoothLLM

Builds on: PALM2-Otter FLAN instruction tuning best-of-N sampling as KL-regularized RL approximation reversal curse bidirectional training

Closest papers by meaning

• PiCO: Peer Review in LLMs based on Consistency Optimization (ICLR 2025 · similarity 0.87)
• Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.87)
• RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.87)
• Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.87)
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.87)
• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.86)