Research map / Vision-language and multimodal / Human and embodied modeling
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, Nicolas Carion
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive alignment loss (ablation) | MDETR · CLEVR-Medium | AP: 83.2 -> 99.0 | Improved |
| add soft-token classification loss (ablation) | MDETR · CLEVR-Medium | AP: 87.7 -> 99.0 | Improved |
| add modulated-detection pre-training curriculum (ablation) | MDETR · CLEVR val | QA accuracy: 68.2 -> 99.7 | Improved |
| use specialized head per question type instead of single QA head (ablation) | MDETR · CLEVR val | QA accuracy: 90.1 -> 99.7 | Improved |
| swap ResNet-101 backbone for EfficientNet-B3 | MDETR · RefCOCO val | accuracy: 86.75 -> 87.51 | Improved |
| swap EfficientNet-B3 for larger EfficientNet-B5 | MDETR · RefCOCO/+/g | accuracy | No clear effect |
| swap ResNet-101 for EfficientNet-B5 | MDETR · GQA test-std | accuracy: 61.99 -> 62.45 | Improved |
| swap ResNet-101 for EfficientNet-B3 | MDETR · PhraseCut | M-IoU: 53.1 -> 53.7 | Improved |
| add pre-training on COCO, VG and Flickr30k | MDETR-R101 · Flickr30k Any-Box val | R@1: 78.9 -> 82.5 | Improved |
| combine multiple annotations per image into one training text | MDETR pre-training data · downstream tasks | accuracy | Improved |
| fine-tune on CLEVR-Humans | MDETR · CLEVR-Humans | accuracy: 59.9 -> 81.7 | Improved |
| train on condition A, test on condition B | MDETR · CoGenT | accuracy: 99.8 -> 76.7 | Hurt |
Later papers that compared with this method
18 later accepted papers report MDETR as a baseline in their results tables; 13 of them report a better number on most of the results they share with it.
- Afford-X: Generalizable and Slim Affordance Reasoning for Task-Oriented Manipulation (TPAMI 2026 · better on 6 of 6 shared results)
- Multi-label Instance-level Generalised Visual Grounding in Agriculture (ECCV 2026 · better on 5 of 5 shared results)
- NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · better on 1 of 1 shared result)
- Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension (AAAI 2025 · better on 5 of 5 shared results)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · better on 1 of 3 shared results)
- Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios (NeurIPS 2024 · better on 3 of 3 shared results)
- GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · better on 1 of 1 shared result)
- Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model (CVPR 2024 · better on 1 of 1 shared result)
10 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.88)
- Question Aware Vision Transformer for Multimodal Reasoning (CVPR 2024 · similarity 0.87)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.87)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.87)
- Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
