Research map / Vision-language and multimodal / Multimodal LLMs
Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScoreRS, a Qwen2-VL-7B-based preference-trained scoring model for remote sensing vision-language data quality, used for data selection, GRPO reward, and best-of-N selection.
Problem
Remote sensing VLM training relies on large synthetic or rule-based image-text data of uneven quality, and there is no open automated quality assessment tool for such RS vision-language data.
Approach
Train a value-head scoring model on RS-specific pairwise preference data (image-caption and vision-instruction) with a Bradley-Terry loss and a three-stage progressive recipe; use the score to select top-ranked data, as a reference-based GRPO reward for open-ended answers, and as a best-of-N selector.
Key results
- ScoreRS top-30% filtering yields RemoteCLIP average classification top-1 of 70.97 vs 69.98 for CLIP-score filtering and 63.52 for full data.
- Qwen2VL-FT with ScoreRS-P30S60 achieves 83.23 average RS classification vs 79.49 for CLIP-P30S60 and 79.15 for full fine-tuning.
- Qwen2VL-7B-RS-R1 with ScoreRS as BoN selector pushes LHRS-Bench accuracy above 70% for the first time, per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rank and select the top 30% of RemoteCLIP data with ScoreRS instead of CLIP-Score | CLIP-ViT-L/14 fine-tuning · RemoteCLIP classification | Avg. top-1 accuracy: 69.98 -> 70.97 | Helped |
| Rank and select the top 30% of RemoteCLIP data with ScoreRS instead of CLIP-Score | CLIP-ViT-L/14 fine-tuning · RemoteCLIP retrieval | Avg. R@1: 26.36 -> 27.11 | Helped |
| Select the top 30% of image-caption data using ScoreRS rather than train on all data | Qwen2VL-7B fine-tuning · LHRS-Bench | accuracy: 64.78 -> 66.24 | Helped |
| Select the top 30% of image-caption data and top 60% of instruction data using ScoreRS rather than CLIP scores | Qwen2VL-7B fine-tuning · VG-DIOR | accuracy: 53.01 -> 55.58 | Helped |
| Select the top 30% of image-caption data and top 60% of instruction data using ScoreRS rather than CLIP scores | Qwen2VL-7B fine-tuning · LHRS-Bench | accuracy: 66.03 -> 66.58 | Helped |
| Use Qwen2VL-2B rather than Qwen2VL-7B as the scoring-model initialization (ablation) | ScoreRS preference scoring · Held-out preference set | accuracy: 82.09 -> 75.58 | Hurt |
| Initialize the scorer with an RS-specific VLM fine-tuned on the complete VHM dataset rather than Qwen2VL-7B (ablation) | ScoreRS filtering and Qwen2VL fine-tuning · VG-DIOR | accuracy: 55.58 -> 53.47 | Hurt |
| Initialize the scorer with an RS-specific VLM fine-tuned on ScoreRS-filtered data rather than Qwen2VL-7B (ablation) | ScoreRS filtering and Qwen2VL fine-tuning · VG-DIOR | accuracy: 55.58 -> 56.21 | Helped |
| Use ScoreRS-based rewards for open-ended questions in RL | Qwen2VL-7B-RS reinforcement learning · VG-DIOR and LHRS-Bench | accuracy | Helped |
| Use ScoreRS rather than CLIP-Score as the best-of-N answer selector | VLM-generated answers · VG-DIOR and LHRS-Bench | accuracy | Helped |
| Use tanh or sigmoid normalization after the value head rather than raw scores (ablation) | ScoreRS training · Preference scoring | training loss | Hurt |
| Use softplus normalization after the value head rather than raw scores (ablation) | ScoreRS training · Preference scoring | training loss | Hurt |
Surprising findings
- Fine-tuning CLIP on the full Skyscript dataset performs worse than the original CLIP model, attributed to redundancy in rule-based captions.
- Initializing the scoring model with an RS-specific VLM (fine-tuned on unfiltered VHM data) gave worse filtering and BoN performance than the general Qwen2-VL-7B initialization.
Limitations
- Preference labels rely on GPT-4o and Qwen2-VL-72B judges; human agreement is 92.6% on a 1,000-pair sample and 90.06%/96.80% on the validation set, so label noise remains.
- BoN evaluation on VG-DIOR uses only the first 1,000 instances due to computational constraints.
- Policy models for preference generation were chosen for diversity under cost constraints and their effect on the scorer is not studied.
Experimental setup
Datasets: LHRS-Align GeoChat LHRS-Instruct SkysenseGPT UltraFeedback RLHF-V RemoteCLIP VHM Skyscript
Benchmarks: NWPU AID fMoW EuroSAT SIRI-WHU WHU-RS19 METERML UCM RSICD RSVQA (HR/LR) VG-DIOR LHRS-Bench
Metrics: top-1 accuracy top-5 accuracy recall@1 recall@5 accuracy IoU
Baselines: CLIP-score filtering LongCLIP-L SigLIP-2 CLIP-LAION-RS GPT-4o reward DPO LLaVA-1.6-7B InternVL-2.5-8B GeoChat VHM SkysenseGPT LHRS-Bot-Nova
Builds on: GRPO DeepSeek-R1 Bradley-Terry model RemoteCLIP LHRS-Bot-Nova Qwen2-VL
Closest papers by meaning
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.91)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.91)
- VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.91)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.90)
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.89)
- CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.