Research map / Detection and segmentation / Open-set and few-shot detection
Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mars-Bench, a 20-dataset benchmark spanning classification, segmentation, and object detection on Mars orbital and rover imagery, with baselines from ImageNet/COCO-pretrained CNN and transformer models, EO foundation models, and proprietary VLMs.
Problem
Mars science lacks standardized, ML-ready benchmarks and evaluation protocols, which has limited the development and fair comparison of foundation models for Martian imagery tasks.
Approach
Curates and standardizes 20 existing or newly built Mars datasets (craters, cones, boulders, frost, landslides, dust devils, terrain, surface classes) into a unified format with fixed splits, few-shot and partitioned variants, and multi-format annotations, then benchmarks many architectures and training regimes with multi-seed IQM-based reporting.
Key results
- YOLO11 gives the best object detection performance across all three detection datasets and in aggregate.
- U-Net achieves the highest segmentation performance, outperforming SegFormer and DPT on nearly all datasets, while ViT-L/16 and SwinV2-B are competitive in classification.
- ImageNet-pretrained ViT-L/16 outperforms EO foundation models (SatMAE, CROMA, Prithvi) on Mars classification tasks; Gemini and GPT-4o Mini show inconsistent zero-shot performance (e.g., 0.84 and 0.70 F1 on mb-mars_seg_msl, but 0.30-0.32 F1 on mb-domars16k).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the training-set size | Mars-Bench classification and segmentation models · Mars-Bench classification and segmentation datasets | normalized test metric | Helped |
| Use ViT-L/16 or SwinV2-B instead of smaller convolutional models | Mars-Bench classification models · Mars-Bench classification datasets | normalized F1-score | Helped |
| Use U-Net instead of transformer-based segmentation models | Mars-Bench segmentation models · Mars-Bench segmentation datasets | normalized IoU | Helped |
| Use YOLO11 instead of other evaluated detection architectures | Mars-Bench object detection models · Three object detection datasets | normalized mAP | Helped |
| Use ImageNet-pretrained ViT-L/16 instead of EO-pretrained SatMAE, CROMA, or Prithvi | Mars-Bench classification models · Selected Mars-Bench classification tasks | normalized F1-score | Helped |
| Use Prithvi rather than other evaluated EO foundation models | Mars-Bench classification models · Selected Mars-Bench classification tasks | normalized F1-score | Hurt |
| Use Gemini 2.0 Flash rather than GPT-4o Mini | Mars-Bench VLM evaluation · mb-domars16k | F1-score: 0.30 vs 0.32 | Mixed |
| Use Gemini 2.0 Flash rather than GPT-4o Mini | Mars-Bench VLM evaluation · mb-surface_cls | F1-score: 0.41 vs 0.44 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o Mini | Mars-Bench VLM evaluation · mb-frost_cls | F1-score: 0.54 vs 0.55 | Helped |
| Use GPT-4o Mini rather than Gemini 2.0 Flash | Mars-Bench VLM evaluation · mb-atmospheric_dust_cls_edr | F1-score: 0.50 vs 0.56 | Helped |
| Use GPT-4o Mini rather than Gemini 2.0 Flash | Mars-Bench VLM evaluation · mb-crater_multi_seg | F1-score: 0.41 vs 0.51 | Helped |
| Use Gemini 2.0 Flash rather than GPT-4o Mini | Mars-Bench VLM evaluation · mb-mars_seg_msl | F1-score: 0.70 vs 0.84 | Helped |
Surprising findings
- ImageNet-pretrained ViT-L/16 beat Earth-observation foundation models pre-trained on satellite data, which the authors attribute to pre-training scale and diversity.
- Proprietary VLMs performed poorly on fine-grained Mars features such as crater types and landmarks despite reasonable results on terrain classes.
Experimental setup
Datasets: mb-atmospheric_dust_cls_edr mb-atmospheric_dust_cls_rdr mb-change_cls_ctx mb-change_cls_hirise mb-domars16k mb-frost_cls mb-landmark_cls mb-surface_cls mb-surface_multi_label_cls mb-boulder_seg mb-conequest_seg mb-crater_binary_seg
Benchmarks: Mars-Bench
Metrics: F1-score IoU mAP accuracy normalized score IQM with bootstrapped confidence intervals
Baselines: ResNet101 SqueezeNet1.1 InceptionV3 SwinV2-B ViT-L/16 U-Net DeepLabV3+ SegFormer DPT YOLO11 SSD RetinaNet
Builds on: Geo-Bench PANGAEA ImageNet pretraining COCO pretraining
Closest papers by meaning
- REOBench: Benchmarking Robustness of Earth Observation Foundation Models (NeurIPS 2025 · similarity 0.86)
- Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions (NeurIPS 2025 · similarity 0.84)
- Interactive Mars Image Content-Based Search with Interpretable Machine Learning (AAAI 2024 · similarity 0.84)
- Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery (CVPR 2025 · similarity 0.84)
- GeoCore-9B: Towards Geo-Aware Generative Foundation Models in Earth Observation (NeurIPS 2026 · similarity 0.84)
- Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.