Research map / Large language models / Agents, code and math
Building Conversational Artifacts to Enable Digital Assistant for APIs and RPAs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses LLMs at build time to generate utterances, slot-filling questions, and next-best-agent recommendations from OpenAPI specs, so chatbot skills for APIs/RPAs need no LLM at runtime; deployed in IBM Watson Orchestrate.
Problem
Converting APIs and RPAs into conversational chatbot skills at scale requires costly manual creation of intent training data, slot-filling questions, and next-action logic, and runtime LLM use is expensive.
Approach
Use LLMs offline in a human-guided "build" phase to augment API descriptions into training utterances, generate attribute questions, and rank next-best agents, with the deployed system relying only on pre-generated artifacts and a trained classifier at run time.
Key results
- Utterance augmentation with LLMs improved Watson Assistant intent accuracy on SGD from 0.55 (baseline) to 0.71 (FLAN-T5-XXL) and on HR from 0.53 to 0.66.
- Llama-2-13b-Chat achieved the highest question generation accuracy (0.797) against ground-truth questions on SGD.
- Next-best-agent approach reached NBA accuracy 0.26 vs 0.10 for Elastic Search on ABCD; the system reported a 77% reduction in build effort per skill (35 minutes saved).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Augment intent-classifier training data with FLAN-T5-XXL-generated utterances | Watson Assistant · HR | accuracy: 0.53 -> 0.66 | Helped |
| Augment intent-classifier training data with LLM-generated utterances | Watson Assistant · SGD | accuracy: 0.35 -> 0.71 | Helped |
| Augment intent-classifier training data with FLAN-T5-XXL-generated utterances | XGBoost · HR | accuracy: 0.04 -> 0.52 | Helped |
| Augment intent-classifier training data with LLM-generated utterances | XGBoost · SGD | accuracy: 0.02 -> 0.51 | Helped |
| Use FLAN-T5-XXL rather than FLAN-T5-XL for question generation | Question generation · SGD | QG accuracy: 0.734 -> 0.719 | Hurt |
| Use Llama-2-13b-Chat rather than Llama-2-13b for question generation | Question generation · SGD | QG accuracy: 0.609 -> 0.797 | Helped |
| Use semantic search with input-output match ranking instead of Elastic Search | Next Best Agent recommendation · ABCD | NBA accuracy: 0.10 -> 0.26 | Helped |
| Use semantic search with input-output match ranking instead of Elastic Search | Next Best Agent recommendation · ABCD | Agent recall: 0.27 -> 0.50 | Helped |
| Use semantic search with input-output match ranking instead of Elastic Search | Next Best Agent recommendation · HR | NBA accuracy: 0.40 -> 0.39 | Hurt |
| Use semantic search with input-output match ranking instead of Elastic Search | Next Best Agent recommendation · HR | Agent recall: 0.58 -> 0.61 | Helped |
Surprising findings
- Larger LLMs did not consistently improve question generation; FLAN-T5-XL (3B) outperformed FLAN-T5-XXL (11B).
- On the HR dataset, Elastic Search matched the proposed NBA approach (agent recall 0.58 vs 0.61), since HR API descriptions have high keyword overlap.
Limitations
- Validation done without the build persona's manual edits, so reported performance is a lower bound on the deployed system.
- Question generation accuracy relies on a human-annotated subset (~15%) to calibrate thresholds.
- Internal HR dataset is not publicly available, limiting reproducibility.
Experimental setup
Datasets: SGD HR (internal) ABCD
Benchmarks: SGD HR ABCD
Metrics: accuracy F1-score fallback rate incorrect rate QG-accuracy NBA accuracy agent recall
Baselines: Baseline (description-only training) Elastic Search (BM25) for next best agent
Builds on: Rizk et al. 2020 multi-agent orchestrator framework OpenAPI specification
Closest papers by meaning
- ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models (ICLR 2025 · similarity 0.86)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.85)
- Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs (NeurIPS 2024 · similarity 0.85)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.85)
- Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent (AAAI 2025 · similarity 0.85)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.