Research map / Large language models / Retrieval and knowledge
UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UrbanKGent, an LLM agent framework that builds urban knowledge graphs via knowledgeable instruction generation and tool-augmented iterative trajectory refinement distilled from GPT-4, then fine-tunes Llama 2/3 models (7/8/13B) with LoRA.
Problem
Urban knowledge graph construction (relational triplet extraction and geospatial knowledge graph completion) still relies on manual rules and effort, and LLMs lack urban heterogeneous relation understanding and geospatial computing ability.
Approach
Inject heterogeneous (spatial/temporal/functional) urban knowledge and geospatial relation definitions with coordinates into instructions, augment GPT-4 reasoning trajectories with self-programmed geospatial tools and iterative verifier-updater self-refinement, then hybrid-instruction fine-tune open Llama models so they run UrbanKGC without GPT API cost.
Key results
- UrbanKGent-13B improves over GPT-4 under the same inference pipeline by 15.56% (GPT-judged RTE accuracy, NYC: 0.52 vs 0.45) and by 11.90% to 14.89% on other NYC metrics.
- The abstract claims UrbanKGent family outperforms 31 baselines and GPT-4 by more than 10% at about 20x lower cost; reported latency/cost comparison shows roughly 20x reduction for 13B vs GPT-4.
- Spearman correlation between GPT-4 and human evaluation is above 0.8 across backbones and above 0.85 across paradigms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the knowledgeable instruction template (ablation) | UrbanKGent-7B · NYC RTE | GPT-evaluated accuracy: 0.38 -> 0.46 | Helped |
| Add the knowledgeable instruction template (ablation) | UrbanKGent-7B · NYC KGC | GPT-evaluated accuracy: 0.42 -> 0.49 | Helped |
| Use multi-view design for RTE (ablation) | UrbanKGent-7B · NYC RTE | GPT-evaluated accuracy: 0.34 -> 0.46 | Helped |
| Use multi-view design for RTE (ablation) | UrbanKGent-7B · NYC KGC | GPT-evaluated accuracy: 0.45 -> 0.49 | No clear effect |
| Add external geospatial tool invocation (ablation) | UrbanKGent-7B · NYC RTE | GPT-evaluated accuracy: 0.45 -> 0.46 | Helped |
| Add external geospatial tool invocation (ablation) | UrbanKGent-7B · NYC KGC | GPT-evaluated accuracy: 0.40 -> 0.49 | Helped |
| Add iterative trajectory self-refinement (ablation) | UrbanKGent-7B · NYC RTE | GPT-evaluated accuracy: 0.44 -> 0.46 | Helped |
| Add iterative trajectory self-refinement (ablation) | UrbanKGent-7B · NYC KGC | GPT-evaluated accuracy: 0.47 -> 0.49 | Helped |
| Use UrbanKGent-13B rather than GPT-4 with the same inference pipeline | UrbanKGC · NYC RTE | GPT-evaluated accuracy: 0.45 -> 0.52 | Helped |
| Use UrbanKGent-13B rather than GPT-4 with the same inference pipeline | UrbanKGC · NYC KGC | GPT-evaluated accuracy: 0.48 -> 0.56 | Helped |
| Use UrbanKGent-13B rather than GPT-4 with the same inference pipeline | UrbanKGC · CHI RTE | GPT-evaluated accuracy: 0.46 -> 0.53 | Helped |
| Use UrbanKGent-13B rather than GPT-4 with the same inference pipeline | UrbanKGC · CHI KGC | GPT-evaluated accuracy: 0.41 -> 0.48 | Helped |
Surprising findings
- Zero-shot GPT-4 performs poorly on UrbanKGC tasks (about 0.38 to 0.48 accuracy), and in-context learning gives limited gains.
- Repeating GPT-4 self-evaluation reduces its correlation with human judgments for RTE (98.56% with one repeat vs 85.27% with ten repeats).
Limitations
- Evaluation relies on GPT-4 self-evaluation for the full data, with human evaluation on only 200 samples.
- Small instruction set (232 and 122 records) used for fine-tuning and evaluation limited to two cities (NYC and Chicago).
- Geospatial relations limited to five RCC candidates and eight self-programmed tools.
Experimental setup
Datasets: NYC-Instruct CHI-Instruct NYC CHI NYC-Large CHI-Large
Benchmarks: NYC RTE/KGC test set CHI RTE/KGC test set NYC-Large CHI-Large
Metrics: accuracy GPT-4 confidence score
Baselines: KG-BERT KG-T5 RelationPrompt PRGC Vicuna-7B Alpaca-7B Mistral-7B Llama-2-7B Llama-3-8B Llama-2-13B Llama-2-70B Llama-3-70B
Builds on: FireAct chain-of-thought LoRA GeoLM
Closest papers by meaning
- Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs (NeurIPS 2024 · similarity 0.85)
- CitySTAR: Agent-Driven Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Grounding (NeurIPS 2026 · similarity 0.85)
- Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.85)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.85)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.85)
- GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.