Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Poly-Autoregressive Prediction for Modeling Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Poly-Autoregressive (PAR) modeling, a small Llama-style transformer that predicts an ego agent's future states from its own history plus other agents' tokens, evaluated on social action forecasting, nuScenes car trajectories, and DexYCB object pose forecasting.
Problem
Predicting an agent's future behavior in multi-agent physical interactions, where single-agent autoregressive models ignore the influence of other interacting agents.
Approach
Represent all agents' states as a flattened token sequence, train with same-agent next-timestep prediction plus a learned agent ID embedding and joint loss over all agents, so the ego agent's future is predicted conditioned on other agents' states.
Key results
- On AVA action forecasting, 2-agent PAR reaches 42.6 mAP vs 40.7 for 1-agent AR (+1.9), and +3.5 on 2-person classes.
- On nuScenes, 3-agent PAR with acceleration tokens and location positional encoding reduces ADE/FDE from 1.50/3.64 (AR, velocity) to 1.35/3.34.
- On DexYCB, 2-agent PAR cuts translation MSE from 3.68e-3 to 2.17e-3 (41% relative) and rotation GEO from 0.919 to 0.837 rad (8.9% relative).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition prediction on another agent using PAR instead of single-agent AR | AVA action forecasting · AVA validation | mAP: 40.7 -> 42.6; +1.9 mAP | Helped |
| condition prediction on the interacting agent using PAR | AVA 2-person action forecasting · AVA 2-person action classes | mAP: +3.5 mAP | Helped |
| use same-agent next-timestep prediction and agent ID embedding instead of multi-agent next-token prediction without agent ID (ablation) | AVA action forecasting · AVA validation | mAP: 38.0 -> 42.6 | Helped |
| use velocity tokens instead of acceleration tokens (ablation) | 3-agent PAR trajectory prediction · nuScenes | ADE: 1.40 -> 1.45 | Hurt |
| use velocity tokens instead of acceleration tokens (ablation) | 3-agent PAR trajectory prediction · nuScenes | FDE: 3.44 -> 3.51 | Hurt |
| add location positional encoding (ablation) | 3-agent PAR trajectory prediction with acceleration tokens · nuScenes | ADE: 1.40 -> 1.35 | Helped |
| add location positional encoding (ablation) | 3-agent PAR trajectory prediction with acceleration tokens · nuScenes | FDE: 3.44 -> 3.34 | Helped |
| condition object pose forecasts on the interacting hand using PAR | Object pose forecasting · DexYCB test set, translation | MSE: 3.68 ⇥10−3 -> 2.17 ⇥10−3; 41% | Helped |
| condition object pose forecasts on the interacting hand using PAR | Object pose forecasting · DexYCB test set, rotation | GEO (rad): 0.919 -> 0.837; 8.9% | Helped |
Surprising findings
- Naive multi-agent next-token prediction (without same-agent next-timestep or agent ID) performs worse than the single-agent AR baseline on AVA (38.0 vs 40.7 mAP).
Limitations
- Relies on high-quality tracking of all agents as input tokens.
- Does not model environment context such as lane maps or images.
- Does not explicitly weigh the relative importance of neighboring agents.
Experimental setup
Datasets: AVA nuScenes DexYCB
Benchmarks: AVA nuScenes DexYCB
Metrics: mAP ADE FDE MSE GEO
Baselines: 1-agent AR Random Token Random Training Trajectory Nearest Neighbor Multiagent NN Mirror
Builds on: Llama MotionLM teacher forcing
Closest papers by meaning
- DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.85)
- AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models (CVPR 2026 · similarity 0.85)
- Motion Forecasting in Continuous Driving (NeurIPS 2024 · similarity 0.85)
- Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation (NeurIPS 2025 · similarity 0.84)
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.84)
- Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.