Research map / Vision-language and multimodal / CLIP and zero-shot

Cracks in the Foundation: A Civil Infrastructure Dataset to Challenge Vision Foundation Models

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Open-vocabulary alignment from VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Cracks in the Foundation (CiF), a ~150k-image, high-resolution civil infrastructure instance segmentation dataset, and shows that supervised detectors plateau near 25% mask mAP while zero-shot foundation models and VLMs perform poorly.

Problem

Pixel-level instance segmentation of civil infrastructure defects (cracks, spalling, rust, algae) is data-scarce and hard for current models, which lack a large, high-resolution benchmark and fail on textureless, thin, non-convex targets.

Approach

A large, native-resolution, expert-annotated instance segmentation dataset for bridge and concrete infrastructure, released in Full and Tiled variants, used to benchmark supervised detectors, promptable segmentation foundation models (SAM3) and VLMs (Qwen3-VL) and expose their failures.

Key results

• EoMT with DINOv3 reaches 24.2 mask mAP50:95 and 32.9 bbox mAP, the best supervised segmentation result.
• SAM3 with oracle bounding boxes reaches 34.2 mAP, while text prompting gives only 2.0 mAP.
• Qwen3-VL-235B-A22B reaches only 2.4 mAP for zero-shot detection on CiF.

Surprising findings

• Zero-shot SAM3 with text prompts and Qwen3-VL score near zero, even the 235B model.
• Supervised crack segmentation stays in single digits in mask AP despite overall detection mAP near 35%.

Limitations

• Focuses on European bridges and concrete; limited coverage of steel or underground structures.
• Defect annotation is subjective, with label variability even among experts.
• Long-tailed class distribution affects rare categories such as Net-Crack and Precipitation.

Experimental setup

Datasets: CiF

Benchmarks: CiF

Metrics: mAP50:95 (mask) mAP50:95 (bbox) AP50:95 per class

Baselines: Mask DINO YOLOv11 YOLOv26 RF-DETR EoMT SAM3 Qwen3-VL

Builds on: Mask DINO EoMT RF-DETR SAM DINOv2

Closest papers by meaning

• CrackSSM: Reviving SSMs for Crack Segmentation via Dynamic Scanning (CVPR 2026 · similarity 0.87)
• SCSegamba: Lightweight Structure-Aware Vision Mamba for Crack Segmentation in Structures (CVPR 2025 · similarity 0.85)
• Meta-Learning Convolutional Neural Architectures for Multi-Target Concrete Defect Classification With the COncrete DEfect BRidge IMage Dataset (CVPR 2019 · similarity 0.84)
• Generalizable Disaster Damage Assessment via Change Detection with Vision Foundation Model (AAAI 2025 · similarity 0.82)
• Towards Open-Vocabulary Industrial Defect Understanding with a Large-Scale Multimodal Dataset (CVPR 2026 · similarity 0.82)
• Mind Marginal Non-Crack Regions: Clustering-Inspired Representation Learning for Crack Segmentation (CVPR 2024 · similarity 0.81)