Research map / Large language models / Agents, code and math
TaskLAMA: Probing the Complex Task Understanding of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TaskLAMA, a human-annotated probe for structured complex task decomposition (steps plus temporal dependencies), with fairer node/edge metrics, and evaluates PaLM 62B with ICL, soft-prompt tuning, sample-and-filter and multi-sequence methods against non-LLM baselines.
Problem
Decomposing complex real-world tasks into steps forming a directed acyclic task graph with temporal dependencies is an important but hard commonsense problem; prior metrics are gameable by duplicate steps and prior datasets have quality issues.
Approach
A human-annotated dataset of 1612 complex tasks with context-specific steps and temporal dependency graphs, plus deduplication-robust Hungarian-matching node metrics and neighborhood-overlap edge metrics; LLM-based generation (ICL, soft-prompt tuning, sample-and-filter, multi-sequence) is compared against heuristic, co-occurrence and hierarchy baselines.
Key results
- ICL with PaLM 62B gives 15% relative improvement in Hungarian F1 and 280% in Rouge2 F2 over the best non-LLM baseline.
- Combining MultSeq, sample-and-filter and soft-prompt tuning yields 7%-37% improvement over the base ICL model depending on the metric.
- Soft-prompt tuning raises temporal dependency accuracy on golden graphs from 47.5 (ICL) to 78.6, while LLM linear ordering remains a strong baseline for generated graphs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ICL instead of the best non-LLM baseline | PaLM 62B task-step generation · TaskLAMA test | Rouge2 F2: 280% relative improvement | Helped |
| Apply Soft-Prompt Tuning instead of ICL (ablation) | PaLM 62B task-step generation · TaskLAMA test | Rouge1 F1: 33.1 -> 38.7 | Helped |
| Add Sample and Filter to ICL (ablation) | PaLM 62B task-step generation · TaskLAMA test | Rouge1 F1: 33.1 -> 36.6 | Helped |
| Combine multiple ICL sequences (ablation) | PaLM 62B task-step generation · TaskLAMA test | Rouge1 F2: 31.7 -> 37.2 | Helped |
| Combine multiple sequences with Soft-Prompt Tuning (ablation) | PaLM 62B task-step generation · TaskLAMA test | Rouge2 F2: 12.3 -> 15.0 | Helped |
| Combine Sample and Filter with Soft-Prompt Tuning (ablation) | PaLM 62B task-step generation · TaskLAMA test | Hungarian F1: 42.4 -> 43.8 | Helped |
| Add task context to the ICL prompt (ablation) | PaLM 62B task-step generation · TaskLAMA test, contextualized tasks | step-generation metrics | Helped |
| Use Soft-Prompt Tuning rather than ICL for pairwise temporal dependency prediction (ablation) | PaLM 62B · TaskLAMA test, golden nodes and edges | accuracy: 47.5 -> 78.6 | Helped |
| Add chain-of-thought rationales to ICL edge prediction (ablation) | PaLM 62B · TaskLAMA test, golden nodes and edges | accuracy: 47.5 -> 49.6 | No clear effect |
| Add SPT edge predictions to the generated linear order (ablation) | PaLM 62B edge prediction, 1 sequence · TaskLAMA test, generated graph | Step Proximity Rouge1: 20.2 -> 20.0 | Hurt |
| Use SPT to predict edges over 2 generated sequences instead of 1 (ablation) | PaLM 62B edge prediction · TaskLAMA test, generated graph | In-Degree Rouge1: 18.0 -> 13.1 | Hurt |
| Use the majority-class no-dependency baseline instead of ICL | Temporal dependency prediction · TaskLAMA test, golden nodes and edges | accuracy: 47.5 -> 53.8 | Helped |
Surprising findings
- Duplicate steps can arbitrarily inflate the standard one-to-many precision and recall metrics, motivating one-to-one Hungarian matching.
- Off-the-shelf LLM pairwise temporal dependency prediction (ICL, CoT) performs near or below majority-class baseline accuracy.
Limitations
- Task graphs are unconditional and single-granularity; real tasks are conditional.
- Similarity-based metrics fail to capture entailment between steps.
- Steps and edges are generated independently rather than jointly; results from a single run due to cost.
Experimental setup
Datasets: TaskLAMA MSComplexTasks ProScript
Benchmarks: TaskLAMA
Metrics: Rouge1 Rouge2 RougeL Hungarian F1/F2 Relaxed Hungarian F1/F2 In-Degree Rouge Out-Degree Rouge Step Proximity Rouge accuracy
Baselines: Repeat Task Repeat Similar Search Query Co-occurrence Search Query Hierarchy ICL with CoT Linear Order LLM Scoring
Builds on: LAMA chain-of-thought prompting in-context learning prompt tuning
Closest papers by meaning
- CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning (ICLR 2025 · similarity 0.87)
- Tree-Planner: Efficient Close-loop Task Planning with Large Language Models (ICLR 2024 · similarity 0.87)
- THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.87)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.86)
- Reasoning Scaffolding: Distilling the Flow of Thought from LLMs (ICLR 2026 · similarity 0.86)
- Active Task Disambiguation with LLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.