AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Xin Yue Li, Jeffrey Bigham, Amy Pavel

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Human perception modeling · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train slide element recognizers on synthetic slides instead of equal-sized human-annotated dataslide element recognition · Slide element recognitionaverage mAP (%): 44.3 -> 55.3Improved
pretrain UI element recognizers on synthetic UIs before finetuning on 50 human samplesUI element recognition · UI element recognition (50 samples)average mAP (%): 47.2 -> 52.4Improved
pretrain on synthetic UIs before finetuning on the full human-annotated setUI element recognition · UI element recognition (full set)average mAP (%): 78.8 -> 81.8Improved
scale synthetic slides for element recognition to twice the human set (ablation)slide element recognition · Slide element recognitionaverage mAP (%): 55.3 -> 64.4Improved
fine-tune VLM on synthetic slides for classification instead of zero-shotLLaVA / Qwen-VL · Slide classificationaverage accuracy (%): 46.8 -> 52.2Improved
fine-tune VLM on small human-annotated data instead of zero-shotLLaVA / Qwen-VL · Slide classificationaverage accuracy (%): 46.8 -> 33.8Hurt
fine-tune VLM on synthetic UIs for classification instead of zero-shotLLaVA / Qwen-VL · UI classificationaverage accuracy (%): 38.9 -> 42.3Improved
fine-tune VLM on small human-annotated UI data instead of zero-shotLLaVA / Qwen-VL · UI classificationaverage accuracy (%): 38.9 -> 31.0Hurt
increase synthetic classification data beyond about 5K samples (ablation)LLaVA / Qwen-VL · Slide and UI classificationaccuracyNo clear effect
scale synthetic slides for classification from equal size to 5,033 (ablation)LLaVA / Qwen-VL · Slide classificationaverage accuracy (%): 52.2 -> 67.9Improved
fine-tune LLaVA on synthetic captions instead of existing caption datasetsLLaVA-1.5-13B · Slide captioningwin rate (%): 77.9% win rateImproved
synthetic-trained open model versus GPT-4V for captioningLLaVA-1.5-13B · Slide captioningwin rate (%): 31.6% win rateHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy