Research map / Vision-language and multimodal / Human and embodied modeling

ConText: Driving In-context Learning for Text Removal and Segmentation

ICML 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts visual in-context learning (MAE-style in-painting) to OCR, proposing task-chaining prompts (image-removal-segmentation), context-aware aggregation, and self-prompting to build the ConText generalist for text removal and segmentation.

Problem

Existing visual in-context learning generalists use single-step image-label compositing, which is single-task and context-free and suffers from visual heterogeneity in text, so they underperform on OCR pixel tasks like text removal and segmentation.

Approach

Chain text removal and segmentation into one prompt (image, erased image, mask) so masked reconstruction covers both task labels; add a cross-attention context-aware aggregation to enrich label features; and periodically use the query itself as its demonstration (self-prompting) to keep in-context learnability for heterogeneous text.

Key results

• ConText achieves PSNR 38.36 and FID 11.04 averaged across three text removal benchmarks, and average fgIoU 76.77% across four segmentation datasets, outperforming V-ICL generalists like Painter and SegGPT.
• ConTextV reaches fgIoU 81.21 on HierText and 89.74 on TextSeg, and PSNR 40.83 on SCUT-EnsText, surpassing task-specific specialists such as Hi-SAM and ViTEraser on several metrics.
• On the PromptText user-prompt benchmark (not trained on), ConTextV obtains PSNR 38.14 and fgIoU 59.19 versus 24.11 PSNR for Painter-based removal and 44.69 fgIoU for SegGPT.

Measured effects reported in the paper

Surprising findings

• With ground-truth demonstrations the proposed modules yield little gain (self-prompting and CAA show only marginal RS-to-GT differences in the ablation), implying random demonstrations push models toward task-specific behavior.
• Multi-demonstration inference gives only modest gains (+0.62 fgIoU at 5 shots), unlike larger gains reported in natural-image V-ICL, attributed to heterogeneous text.

Limitations

• Gains from self-prompting and CAA diminish when given ground-truth demonstrations, so the model's upper-bound in-context reasoning remains limited.
• Synthetic-natural domain gap: performance on SCUT-Syn is not best among specialists; ConTextV's removal labels for TextSeg and segmentation labels for SCUT-EnsText are machine-generated and noisy.
• Multi-demonstration and double inference bring only small gains at added labeling or compute cost.

Experimental setup

Datasets: HierText TextSeg TotalText SCUT-EnsText SCUT-Syn FST CLWD

Benchmarks: HierText TotalText FST TextSeg SCUT-EnsText SCUT-Syn PromptText CLWD

Metrics: PSNR MSSIM MSE AGE pEPs pCEPs FID fgIoU F-score

Baselines: MAE-VQGAN Painter SegGPT ViTEraser Hi-SAM SegFormer DeepLabV3+ HRNetV2-W48+OCR TFT EAFormer UPOCR Pix2Pix

Builds on: Painter SegGPT MAE ViTEraser Hi-SAM

Closest papers by meaning

• Towards More Unified In-context Visual Understanding (CVPR 2024 · similarity 0.88)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.87)
• Visual In-Context Prompting (CVPR 2024 · similarity 0.87)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.87)
• Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.86)
• Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.86)