Research map / Multi-agent systems and games / Human-AI and goal-conditioned
CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAML, a multi-agent auxiliary modality learning framework where agents share multi-modal data during training via a teacher and distill into an RGB-only student for test-time collaborative decision-making and segmentation.
Problem
Multi-modal learning systems often lack some modalities at inference, and existing auxiliary modality learning is limited to single-agent settings, while multi-agent collaboration work ignores missing modalities at test time.
Approach
Train a multi-agent teacher with all modalities shared across agents, then distill via knowledge distillation into a student where each agent uses only a reduced modality subset (RGB), with embeddings shared among agents at inference.
Key results
- CAML achieves up to 58.1% mean ADR improvement over AML in the red light violation scenario.
- CAML improves mIoU by 8.9% (indoor) and 10.6% (outdoor) over AML on CoPeD aerial-ground segmentation.
- CAML has 1.0 KB package size and 3.7 ms latency, versus 65.5 KB/90.0 ms for COOPERNAUT and 4.9 KB/18.5 ms for STGN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use collaborative multi-agent training and auxiliary LiDAR knowledge via distillation instead of single-agent AML | CAML · CAV overtaking, left turn, and red-light violation | ADR: 13.0%; 34.6%; 58.1% | Helped |
| use CAML rather than COOPERNAUT | CAV decision-making · CAV accident-prone scenarios | ADR: up to 21.5% | Helped |
| use CAML with RGB-only inference rather than STGN with RGBD inference | CAV decision-making · Left Turn | ADR: 10.9% | Helped |
| use CAML rather than AML | Collaborative semantic segmentation · CoPeD Indoor | mIoU: 8.9% | Helped |
| use CAML rather than AML | Collaborative semantic segmentation · CoPeD Outdoor | mIoU: 10.6% | Helped |
| distill from a full-modality teacher instead of training multi-agent BC with RGB only and no KD (ablation) | CAML student · CAV Overtaking | ADR: 85.2\ensuremath{\pm}1.0 -> 92.8\ensuremath{\pm}0.8 | Helped |
| distill from a full-modality teacher instead of training multi-agent BC with RGB only and no KD (ablation) | CAML student · CAV Left Turn | ADR: 56.4\ensuremath{\pm}1.5 -> 67.3\ensuremath{\pm}1.0 | Helped |
| distill from a full-modality teacher instead of training multi-agent BC with RGB only and no KD (ablation) | CAML student · CAV Red Light Violation | ADR: 55.8\ensuremath{\pm}1.3 -> 66.1\ensuremath{\pm}1.4 | Helped |
| add state information as an auxiliary training modality (ablation) | CAML · CAV Overtaking | ADR: 92.8\ensuremath{\pm}0.8 -> 94.0\ensuremath{\pm}0.7 | Helped |
| retain LiDAR rather than RGB at test time (ablation) | CAML · CAV Red Light Violation | ADR: 66.1\ensuremath{\pm}1.4 -> 67.2\ensuremath{\pm}1.2 | Helped |
| use intermediate feature cooperation rather than late, output-level cooperation (ablation) | CAML · CAV Left Turn | ADR: 60.2\ensuremath{\pm}1.3 -> 67.3\ensuremath{\pm}1.0 | Helped |
| fuse RGB and depth within each robot before sharing fused features (Pre-fusion CAML) (ablation) | CAML semantic segmentation · CoPeD Indoor and Outdoor | mIoU: CAML: 60.1, 66.8; Pre-fusion CAML: 59.2%, 65.8% | No clear effect |
Surprising findings
- CAML with RGB-only testing exceeds STGN (which uses RGB and depth at test time) by 10.9% mean ADR in the left-turn scenario.
Limitations
- Current student model does not dynamically vary modalities over time; modality dropout is proposed as future work.
- Doubles training parameters and time per epoch relative to AML (39.3M vs 19.5M parameters).
- Information-gain analysis is intuitive rather than a formal proof.
Experimental setup
Datasets: AUTOCASTSIM CoPeD
Benchmarks: AUTOCASTSIM (overtaking, left turn, red light violation) CoPeD (indoor NYUARPL, outdoor HOUSEA)
Metrics: Accident Detection Rate (ADR) Expert Imitation Rate (EIR) mIoU package size (KB) latency (ms)
Baselines: AML COOPERNAUT STGN FCN Late Cooperation RGB-only BC
Builds on: Auxiliary Modality Learning knowledge distillation COOPERNAUT STGN
Closest papers by meaning
- Towards Robust Multi-Modal Reasoning via Model Selection (ICLR 2024 · similarity 0.86)
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.85)
- DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.85)
- AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning (AAAI 2026 · similarity 0.85)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.84)
- STAMP: Scalable Task- And Model-agnostic Collaborative Perception (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.