Research map / Large language models / Retrieval and knowledge
Can AI Assistants Know What They Don't Know?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds model-specific "I don't know" datasets from TriviaQA by sampling an assistant's own answers, then aligns Llama-2-7b-chat via prompting, SFT, DPO, BoN, PPO, and HIR so it refuses questions it does not know.
Problem
LLM assistants give confident factual errors on knowledge-intensive questions; they do not reliably recognize and refuse questions outside their own knowledge.
Approach
Label each question as known or unknown to the specific assistant by its empirical accuracy over multiple samples (Ik threshold), build a model-specific IDK dataset with refusal templates and the model's own correct answers, and align the model with it using several training methods; the Truthful rate (Ik-Ik plus Ik-Idk) measures the outcome.
Key results
- Idk-BoN (N=10) reaches the highest TriviaQA Truthful rate of 78.96%, up from 66.93% for Idk-Prompting.
- On ALCUNA (unanswerable questions), Idk-HIR attains 98.96% Ik-Idk rate versus 91.67% for prompting.
- Idk-SFT lowers Ik-Ik rate to 28.57% from 37.36% for prompting, showing over-refusal; preference methods like DPO recover Ik-Ik to 39.30%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align with the model-specific Idk dataset using SFT | Llama-2-7b-chat · TriviaQA | Truthful rate: +7.82 | Helped |
| Align with the model-specific Idk dataset using SFT | Llama-2-7b-chat · Natural Questions | Truthful rate: +8.45 | Helped |
| Use BoN with 10 sampled responses | Llama-2-7b-chat · TriviaQA | Truthful rate: +12.03 | Helped |
| Use HIR with confidence-level-relabelled datasets | Llama-2-7b-chat · Natural Questions | Truthful rate: +10.83 | Helped |
| Use preference optimization instead of SFT | Llama-2-7b-chat · Natural Questions | Truthful rate | Hurt |
| Increase model size from 7B to 13B (ablation) | Idk-SFT · TriviaQA | Truthful rate: +0.60 | Helped |
| Increase model size from 7B to 70B (ablation) | Idk-SFT · TriviaQA | Truthful rate: +5.8 | Helped |
| Use a non-model-specific Idk-Mistral dataset (ablation) | Idk-SFT on Llama-2-7b-chat · TriviaQA | Truthful rate: ↓5.75 | Hurt |
| Use a non-model-specific Idk-Baichuan dataset (ablation) | Idk-SFT on Llama-2-7b-chat · TriviaQA | Truthful rate: ↓12.83 | Hurt |
| Add SFT loss with α=0.01 to DPO (ablation) | Idk-DPO · TriviaQA, Ik-threshold=1.0 | Truthful rate | Helped |
| Add SFT loss to DPO (ablation) | Idk-DPO · TriviaQA, Ik-threshold=0.5 | Truthful rate | Hurt |
| Raise the Ik threshold (ablation) | Idk-SFT · TriviaQA | Truthful rate | Helped |
Surprising findings
- Using an IDK dataset built from another model (Mistral or Baichuan) reduces Truthful rate, so model-specific data matters.
- Validation loss selection led to over-refusal for Llama-2-70b-chat, so the final checkpoint was used instead.
Limitations
- Correctness is judged by lexical matching against gold answers, which may misjudge some responses.
- Truthful rate gains come with reduced Ik-Ik rate under SFT, an alignment tax, and results are measured only on QA-style data.
- Model-specific datasets must be rebuilt for each assistant, limiting transfer.
Experimental setup
Datasets: TriviaQA Natural Questions (NQ-Open) ALCUNA
Benchmarks: TriviaQA Natural Questions ALCUNA
Metrics: Ik-Ik rate Ik-Idk rate Truthful rate
Baselines: Idk-Prompting Idk-SFT Idk-DPO Idk-PPO Idk-HIR
Builds on: DPO PPO RLHF Hindsight Instruction Relabeling Self-Instruct
Closest papers by meaning
- Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs (NeurIPS 2024 · similarity 0.86)
- ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant (AAAI 2025 · similarity 0.84)
- IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.83)
- Building Conversational Artifacts to Enable Digital Assistant for APIs and RPAs (AAAI 2024 · similarity 0.83)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.83)
- Boosting the Potential of Large Language Models with an Intelligent Information Assistant (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.