Research map / Vision-language and multimodal / CLIP and zero-shot
GiT: Towards Generalist Vision Transformer through Universal Language Interface
Haiyang Wang, Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naeem, Hongsheng Li, Bernt Schiele, Liwei Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| joint multi-task training of five tasks instead of single-task training | GiT-B · COCO2017 detection | AP: 45.1 -> 46.7 | Improved |
| joint multi-task training instead of single-task training | GiT-B · ADE20K semantic segmentation | mIoU: 47.7 -> 47.8 | No clear effect |
| joint multi-task training instead of single-task training | GiT-B · COCO Caption | CIDEr: 107.9 -> 112.6 | Improved |
| scale model from ViT-B to ViT-H | GiT multi-task · COCO2017 detection | AP: 46.7 -> 52.9 | Improved |
| universal training on 27 datasets instead of five-dataset multi-task training | GiT-B · Cityscapes semantic segmentation (zero-shot) | mIoU: 34.4 -> 56.2 | Improved |
| universal training instead of multi-task training | GiT-B · DRIVE few-shot (K=5) | mDice: 34.3 -> 51.1 | Improved |
| use decoder-only architecture instead of encoder-decoder (ablation) | GiT-B multi-task · ADE20K semantic segmentation | mIoU: 46.9 -> 47.8 | Improved |
| add modality experts (multiple FFN) (ablation) | GiT-B multi-task · COCO2017 detection | AP: 46.1 -> 46.2 | No clear effect |
| add text conditioning (image-to-text attention) (ablation) | GiT-B multi-task · RefCOCO visual grounding | Acc@0.5: 78.7 -> 85.8 | Improved |
| add new transformer layers on top of SAM-initialized ViT (ablation) | GiT-B single-task · COCO2017 detection | AP: 40.2 -> 45.1 | Improved |
| use accelerated global attention over shared observations only (ablation) | GiT-B single-task · ADE20K semantic segmentation | mIoU and training time: 47.9 -> 47.7; 51h -> 35h | Mixed |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 0 of 2 on most shared results.
- Unified-IO XL — UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks (ICLR 2023) · better on 0 of 1 shared result
- BLIP-2 — BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022) · better on 0 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.88)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.88)
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.88)
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.88)
- xT: Nested Tokenization for Larger Context in Large Images (ICML 2024 · similarity 0.87)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
