Research map / Large language models / AI and society
Data Efficient Paradigms for Personalized Assessment of Black-Box Taskable AI Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Doctoral consortium paper proposing a personalized AI assessment module that queries black-box taskable AI systems via instruction sequences in simulators to learn user-interpretable PDDL models of their capabilities, summarizing prior preliminary versions.
Problem
Laypeople using black-box taskable AI systems cannot easily understand their capabilities and safe operating limits, risking unintended side effects.
Approach
A query-response interface lets the assessment module interrogate a black-box agent, with query generation reduced to planning and top-down elimination of abstract PDDL models, to derive a user-interpretable capability model; capabilities are also discovered from state transitions and described in the user's concept vocabulary.
Key results
- AIA takes less time per query and converges better to the correct model than FAMA in stationary, fully observable, deterministic settings.
- QACE shows few-shot generalization, convergence to a sound and complete model, and greater sample efficiency and accuracy than GLIB for learning lifted relational models in stochastic settings.
- A user study found users answered questions faster and more correctly using discovered capabilities than primitive actions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIA instead of FAMA | Agent interrogation algorithm (AIA) · Stationary, fully observable, deterministic settings | time per query | Helped |
| Use AIA instead of FAMA | Agent interrogation algorithm (AIA) · Stationary, fully observable, deterministic settings | convergence to correct model | Helped |
| Use capability descriptions instead of primitive actions | AI assessment module · User study | time to answer questions | Helped |
| Use capability descriptions instead of primitive actions | AI assessment module · User study | response correctness | Helped |
| Use QACE instead of GLIB | Query-based autonomous capability estimation (QACE) · Stochastic settings | sample efficiency | Helped |
| Use QACE instead of GLIB | Query-based autonomous capability estimation (QACE) · Stochastic settings | model accuracy | Helped |
Limitations
- Mostly summarizes prior preliminary versions; no new experiments reported in this paper.
- Current methods assume fully observable, deterministic settings in early versions; extension to general settings is planned.
- Relies on the AI system supporting a query-response interface and a compatible simulator.
Experimental setup
Metrics: time per query convergence to correct model sample efficiency accuracy of learned model
Baselines: FAMA GLIB
Builds on: AIA (agent interrogation algorithm) DAAISy QACE FAMA GLIB
Closest papers by meaning
- Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.83)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.83)
- Minerva: A Programmable Memory Test Benchmark for Language Models (ICML 2025 · similarity 0.83)
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.82)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.81)
- Scalable AI Safety via Doubly-Efficient Debate (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.