Research map / Vision-language and multimodal / Multimodal LLMs
Non-Natural Image Understanding with Advancing Frequency-based Vision Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FM-ViT, a CLIP ViT with learnable Fourier-based frequency modulation of self-attention outputs (sample-adaptive via wavelet features), and EDGE, an MLLM combining FM-ViT with a CNN encoder, for non-natural image understanding (geometry, charts, functions).
Problem
Multimodal LLMs underperform on non-natural images (geometric figures, charts, function plots) because ViT vision encoders attenuate high-frequency information needed for points, lines, and angles, and non-natural image datasets are costly to collect.
Approach
Decompose ViT self-attention outputs into DC and high-frequency components via Fourier decomposition and re-weight them with learnable coefficients that are modulated per sample by wavelet-transform statistics of the input image; combine with a CNN encoder and a three-stage training pipeline to build EDGE.
Key results
- On GeoQA, the model reaches 68.2% accuracy, above G-LLaVA 7B (64.2%) and EAGLE (67.1%) with the same Vicuna-1.5-7B base.
- On ChartQA the model reaches 72.5%, above ChartLlama (69.6%); on FunctionQA it reaches 43.5%, the best open-source score but below GPT-4V (69.5%).
- Frequency modulation raises encoder classification accuracy by +3.6 points (68.3 to 71.9 with sample adaptation adding +1.2 more to 73.1 overall) per the ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add frequency modulation to the vision encoder (ablation) | CLIP ViT-L/14 · Overall (3 types of data) | Classify: 68.3 -> 73.1 | Helped |
| Add frequency modulation to the vision encoder (ablation) | CLIP ViT-L/14 · Overall (3 types of data) | Txt2Img: 84.3 -> 86.6 | Helped |
| Add frequency modulation to the vision encoder (ablation) | CLIP ViT-L/14 · Overall (3 types of data) | Img2Txt: 84.4 -> 86.9 | Helped |
| Add sample-wise adaptation (ablation) | FM-ViT · Overall (3 types of data) | Classify: 71.9 -> 73.1 | Helped |
| Add sample-wise adaptation (ablation) | FM-ViT · Overall (3 types of data) | Txt2Img: 85.8 -> 86.6 | Helped |
| Add sample-wise adaptation (ablation) | FM-ViT · Overall (3 types of data) | Img2Txt: 86.1 -> 86.9 | Helped |
| Use EDGE with FM-ViT instead of the strongest listed baseline | Vicuna-1.5-7B MLLM · GeoQA | accuracy: 67.1 -> 68.2 | Helped |
| Use EDGE with FM-ViT instead of the strongest listed open-source baseline | FunctionQA · FunctionQA | accuracy: 40.3 -> 43.5 | Helped |
| Use EDGE with FM-ViT instead of ChartLLaMA | ChartQA · ChartQA | accuracy: 69.6 -> 72.5 | Helped |
Surprising findings
- CNN-based ConvNeXt outperforms ViT-B encoders on classification and retrieval for non-natural images, though ViT-L becomes comparable to CNNs.
Limitations
- Classification and retrieval test sets are small and self-annotated (169 geometric, 5 function, 20 chart classes; 1000 caption pairs), and caption scoring relies on an LLM judge.
- Gains on FunctionQA remain well below GPT-4V.
- Frequency analysis is qualitative and based on visualizations and a single high-frequency measurement.
Experimental setup
Datasets: Geo170K MAVIS-Caption MAVIS-Instruct ChartLlama data GeoQA MathVista FunctionQA ChartQA
Benchmarks: GeoQA MathVista FunctionQA ChartQA Geo caption Chart caption Function caption
Metrics: accuracy classification accuracy retrieval accuracy caption correctness and detail scores
Baselines: LLaVA-1.5 LLaMA-Adapter LLaVA-NeXT G-LLaVA MAVIS EAGLE Math-LLaVA ChartLlama Unichart GPT-4V Gemini-1.0-Pro Qwen-VL-Max
Builds on: CLIP LLaVA-1.5 G-LLaVA MAVIS ChartLlama
Closest papers by meaning
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.87)
- Language-guided Frequency Modulation for Large Vision-Language Models (CVPR 2026 · similarity 0.86)
- ViTamin: Designing Scalable Vision Models in the Vision-Language Era (CVPR 2024 · similarity 0.86)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.86)
- Beyond Text: Frozen Large Language Models in Visual Signal Comprehension (CVPR 2024 · similarity 0.86)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.