Research map / Large language models / Retrieval and knowledge
DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DFA-RAG, which learns a definite finite automaton of conversation tags from training dialogues and uses it to route retrieval of in-context examples for LLM response generation in customer-service and task-oriented dialogue.
Problem
LLM conversational agents struggle to produce regulated, workflow-compliant responses in scenarios with predetermined response guidelines (e.g., customer service, emotional support), and standard RAG retrieval does not match partial conversational flows well.
Approach
Model training dialogues as a DFA over LLM-extracted semantic tags (learned via a tag tree with state merging by child-tag similarity), then navigate the DFA with the user's tagged utterances and retrieve dialogues from the matched state as in-context examples for the LLM.
Key results
- On GPT-4, DFA-RAG achieves an average win rate of 74.3 versus naive GPT-4, above the best RAG baseline at 70.6.
- On GPT-3.5, DFA-RAG achieves an average win rate of 75.9 versus 71.6 for RAG.
- On MultiWOZ, DFA-RAG reports Inform 93.3 and Success 90.0, the highest Success among listed methods, without ground-truth dialogue states.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DFA-based retrieval and routing instead of standard RAG | GPT-4 · All domains | win rate: 70.6 -> 74.3 | Helped |
| Use DFA-based retrieval and routing instead of standard RAG | GPT-3.5 · All domains | win rate: 71.6 -> 75.9 | Helped |
| Use DFA-RAG instead of RAG | GPT-4 · AskPlayStation | win rate: 66.3 -> 65.9 | Hurt |
| Use DFA-RAG instead of MarCo | MultiWOZ / CambridgeInfo · MultiWOZ / CambridgeInfo | Inform: 94.5 -> 93.3 | Hurt |
| Use DFA-RAG instead of TOATOD | MultiWOZ / CambridgeInfo · MultiWOZ / CambridgeInfo | Success: 79.8 -> 90.0 | Helped |
| Use DFA-RAG instead of the naive GPT-3.5 base model | GPT-3.5 · AirbnbHelp example dialogue | alignment with ground truth | Helped |
Surprising findings
- Fine-tuning GPT-3.5 (FT-LLM) underperforms naive prompting on most domains, with average win rate 64.2, attributed to limited domain data.
Limitations
- Win-rate evaluation relies on GPT-4 as judge comparing against ground-truth dialogues, with the judge's agreement with humans cited from prior work.
- Tag extraction depends on LLM outputs and heuristic leaf selection and merging thresholds (lambda=0.1, tau=5) chosen by the authors.
- Evaluation is limited to Twitter customer-service and MultiWOZ domains with no human evaluation.
Experimental setup
Datasets: AmazonHelp DeltaSupport AskPlayStation AirbnbHelp NikeSupport CambridgeInfo (MultiWOZ)
Benchmarks: AmazonHelp DeltaSupport AskPlayStation AirbnbHelp NikeSupport CambridgeInfo (MultiWOZ)
Metrics: win rate Inform Success
Baselines: Random Sampling BM25 RAG FT-LLM HDSA MarCo HDNO GALAXY KRLS AuGPT MTTOD RSTOD
Builds on: retrieval-augmented generation semantic router AlpacaEval
Closest papers by meaning
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.87)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.86)
- One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.86)
- RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models (NeurIPS 2025 · similarity 0.86)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.86)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.