Research map / Vision-language and multimodal / Multimodal LLMs
JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds JarvisIR, a VLM agent (Llava-Llama3-8B) that plans tasks and selects expert restoration tools for coupled adverse-weather degradations, trained with SFT on 150K synthetic CleanBench pairs and then an unsupervised-style MRRHF alignment on 80K real images guided by an IQA reward.
Problem
Vision-centric perception in adverse weather involves unknown, coupled degradations; existing task-specific and all-in-one restoration methods fail on real data, and chaining expert restoration tools is sensitive to task order.
Approach
A VLM controller schedules expert restoration tools; SFT on synthetic instruction-response data is followed by MRRHF, an extension of RRHF that uses a hybrid online/offline candidate generation, entropy regularization, score-difference-weighted ranking loss, and a z-score-fused ensemble of VLM/non-VLM IQA models as reward, so it can align on unlabeled real images.
Key results
- JarvisIR-MRRHF reaches 67.25 MUSIQ in night scenes vs. 47.30 for the best all-in-one baseline (AutoDIR), and 70.38 in rain scenes.
- In the decision-making ablation, JarvisIR-MRRHF ranks in the top 4.8% of possible tool-order/model decisions versus 14.3% for JarvisIR-SFT and 18.6% for human experts.
- Hybrid sample generation gives reward 0.67 and diversity 6.55, versus 0.43/3.63 offline and -0.87/1.27 online.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune JarvisIR-SFT on CleanBench-Real using MRRHF | JarvisIR-SFT · CleanBench-Real validation | Score: 5.17 -> 6.21 | Helped |
| Fine-tune JarvisIR-SFT on CleanBench-Real using MRRHF | JarvisIR-SFT · CleanBench-Real validation, night scenes | MUSIQ: 60.77 -> 67.25 | Helped |
| Fine-tune JarvisIR-SFT on CleanBench-Real using MRRHF | JarvisIR-SFT · CleanBench-Real validation, rain scenes | MUSIQ: 65.03 -> 70.38 | Helped |
| Fine-tune JarvisIR-SFT on CleanBench-Real using MRRHF | JarvisIR-SFT · CleanBench-Real validation, fog scenes | MUSIQ: 70.45 -> 74.22 | Helped |
| Add entropy regularization (ablation) | MRRHF · CleanBench-Real validation | Reward: 0.50 -> 0.67 | Helped |
| Add entropy regularization (ablation) | MRRHF · CleanBench-Real validation | Diversity: 4.56 -> 6.55 | Helped |
| Use hybrid rather than offline sample generation (ablation) | MRRHF · CleanBench-Real validation | Reward: 0.43 -> 0.67 | Helped |
| Use hybrid rather than online sample generation (ablation) | MRRHF · CleanBench-Real validation | Reward: -0.87 -> 0.67 | Helped |
| Use hybrid rather than offline sample generation (ablation) | MRRHF · CleanBench-Real validation | Diversity: 3.63 -> 6.55 | Helped |
| Use hybrid rather than online sample generation (ablation) | MRRHF · CleanBench-Real validation | Diversity: 1.27 -> 6.55 | Helped |
| Replace MRRHF with vanilla RRHF (ablation) | JarvisIR-SFT · CleanBench-Real validation | Reward: 0.67 -> 0.40 | Hurt |
| Replace MRRHF with vanilla RRHF (ablation) | JarvisIR-SFT · CleanBench-Real validation | MUSIQ: 71.43 -> 63.89 | Hurt |
Surprising findings
- Pure online sampling in MRRHF collapsed during training, with reward dropping to -0.87 and diversity to 1.27.
- Human expert baseline (18.6% rank) is outperformed by the SFT model, and MRRHF further improves to 4.8%.
Limitations
- Cannot handle restoration scenarios outside the tool set, such as underwater, old photo, or blind face restoration.
- Tools are lightweight and not state-of-the-art, and VLM fixed-resolution input may degrade degradation perception.
Experimental setup
Datasets: CleanBench CleanBench-Real ACDC Q-Instruct
Benchmarks: CleanBench-Real validation set
Metrics: MUSIQ MANIQA CLIP-IQA+ LIQE
Baselines: AirNet AutoDIR DA-CLIP PromptIR MiOIR InstructIR T3-DiffWeather Vanilla RRHF
Builds on: RRHF RLHF self-instruct PRO Llava-Llama3
Closest papers by meaning
- Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.85)
- DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.85)
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.85)
- DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution (NeurIPS 2026 · similarity 0.84)
- Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start (NeurIPS 2025 · similarity 0.84)
- STAMP: Scalable Task- And Model-agnostic Collaborative Perception (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.