Research map / Large language models / AI and society

Models That Know How Evaluations Are Designed Score Safer

NeurIPS 2026 · Large language models › AI and society · approach: Human-centered trust and agency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes Nemotron 49B, Qwen3 32B and GLM 4.7 Flash via LoRA on ~106M tokens of synthetic documents describing evaluation traits, finding higher refusal and lower harmfulness on five safety benchmarks even on evaluation-unaware responses.

Problem

Safety benchmark scores may be inflated by models' parametric knowledge about how evaluations are designed (evaluation meta-knowledge), a confounder distinct from contamination or verbalized evaluation awareness.

Approach

Synthetic document fine-tuning on documents describing evaluation traits (verifiable structure, placeholders, harmful requests, etc.) without asserting facts about the model's own evaluation behavior; safety gains persist after filtering out evaluation-aware responses and exceed FineWeb and type-hint control models.

Key results

• Nemotron 49B refusal on AgentHarm rises from 9.9% to 31.0% (+21.1 pp) after training on evaluation-trait documents.
• Harmfulness on non-refused responses drops by 20.74 points on average for Nemotron 49B traits vs base, versus 8.40 for the type-hints control.
• On evaluation-unaware responses, Nemotron 49B refusal increases by 9.1 on average and Qwen3 32B by 10.7 across the toxic benchmarks; GLM 4.7 Flash shows no clear gain.

Surprising findings

• Safety gains persist on responses where the model does not verbalize evaluation awareness, suggesting an implicit, hard-to-detect confounder.
• GLM 4.7 Flash shows inconsistent safety gains and substantial general capability loss (8.13 points) after traits fine-tuning.

Limitations

• Only models of roughly 30-50B parameters tested, mostly reasoning models.
• Evaluation awareness measured only when explicitly verbalized in responses.
• Unclear whether observed gains reflect score inflation or genuine improved alignment; deployment data scarce.

Experimental setup

Datasets: FineWeb synthetic evaluation-trait documents (106M tokens) ToxicChat

Benchmarks: AgentHarm Triggers StrongREJECT OR-Bench Agentic Misalignment MMLU BBH TruthfulQA

Metrics: refusal rate harmfulness over-refusal accuracy

Baselines: Nemotron 49B type-hints model FineWeb-finetuned Qwen3 32B FineWeb-finetuned GLM 4.7 Flash

Builds on: synthetic document fine-tuning Inspect framework

Closest papers by meaning

• EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.88)
• The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness (NeurIPS 2025 · similarity 0.86)
• AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.86)
• Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.84)
• SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior (ICML 2025 · similarity 0.84)
• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (NeurIPS 2025 · similarity 0.84)