Research map / Large language models / AI and society

Position: Future Research and Challenges Remain Towards AI for Software Engineering

ICML 2025 · Large language models › AI and society · approach: Social-science view of AI evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper that taxonomizes AI software engineering tasks by scope, logical complexity, and human intervention, identifies data, scale, interaction, and measurement challenges, and proposes research directions for AI for code.

Problem

Automated software engineering with AI remains far from full automation; the paper targets the lack of a structured view of SWE tasks and the cross-cutting bottlenecks limiting current code LLMs.

Approach

Proposes a three-measure taxonomy (scope, logical complexity, level of human intervention) over a broad set of SWE tasks beyond code generation, groups current bottlenecks into data, scale, interaction, and measurement, and outlines paths forward in data collection, training, and inference.

Key results

• Proposes a taxonomy of SWE tasks scored on scope (function, unit, project), logical complexity (low to high), and human autonomy level.
• Identifies four cross-cutting challenges: data, scale, interaction, and measurement.
• Argues that scaling alone is insufficient, citing SWE-bench progress from about 2% to 73% as motivation for continued research.

Surprising findings

• On HumanEval, Qwen-2.5 reaches 83% accuracy in Python but only 27% in D, illustrating a low-resource language gap.
• Code generated by RL with test-case rewards can be correct but low quality, since correct but poor code still receives high reward.

Limitations

• No new experiments; claims rest on cited literature and examples.
• Taxonomy and challenge selection are opinionated rather than systematically derived.

Experimental setup

Builds on: DreamCoder DeepSeek-R1 SWE-RL

Closest papers by meaning

• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.82)
• Programming with Pixels: Can Computer-Use Agents do Software Engineering? (ICLR 2026 · similarity 0.82)
• RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph (ICLR 2025 · similarity 0.82)
• Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench (NeurIPS 2026 · similarity 0.81)
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.81)
• Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.81)