Research map / Vision-language and multimodal / Multimodal LLMs

Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HouseMind, an MLLM that tokenizes floor plan outlines and rooms with hierarchical VQ-VAEs and fine-tunes a Qwen3-0.6B backbone to unify floor plan understanding, text-conditioned generation, and editing.

Problem

Floor plan design requires joint reasoning over geometry, semantics, and spatial hierarchy; existing diffusion and VLM approaches lack coherent room-level spatial reasoning, unified multitask support, and local deployability.

Approach

Discretize floor plan outlines and individual room instances into codebook tokens (outline VQ-VAE and outline-conditioned room VQ-VAE), add them to an LLM vocabulary, and train in three stages (embedding init, multimodal pre-training, SFT) so one autoregressive model handles understanding, generation, and editing.

Key results

• Generation: HouseMind-O reaches Micro/Macro IoU 0.710/0.654 and FID 1.89 versus 0.589/0.521 and 11.3 for ChatHouseDiffusion on the shared test set.
• Understanding: HouseMind-U achieves 0.969 LocAcc and 0.990 AdjAcc, versus at most 0.546 LocAcc and 0.597 AdjAcc for the strongest VLM baseline (MiniCPM-V 4.5 AdjAcc).
• Editing: HouseMind-E reaches Node F1 0.998 and GED 0.467 with about 3 s per sample, versus about 240 s for Qwen-Image-Edit-2509 and FLUX.1-Kontext-dev.

Surprising findings

• Structural correction post-processing barely changes pixel metrics (Micro IoU unchanged), improving mainly GED and Edge Overlap.

Limitations

• Editing only supports simple room addition and deletion, not complex topological changes.
• Doors, windows, and furniture are not modeled.
• Layouts are not fully aligned with human design preferences and aesthetic constraints.

Experimental setup

Datasets: RPLAN

Benchmarks: unified floor plan understanding/generation/editing benchmark derived from RPLAN (2,308 test samples)

Metrics: success rate room match rate room location accuracy area difference adjacency accuracy relation accuracy Micro IoU Macro IoU SSIM PSNR

Baselines: LLaVA-v1.6-Mistral-7B-HF Qwen3-VL-8B-Instruct InternVL3.5-8B MiniCPM-V 4.5 Tell2Design ChatHouseDiffusion FloorPlanLLaMA Qwen-Image-Edit-2509 FLUX.1-Kontext-dev GPT-5 Gemini 2.5 Pro

Builds on: VQ-VAE MaskPLAN Tell2Design FloorPlan-LLaMa

Closest papers by meaning

• Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.87)
• Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models (NeurIPS 2026 · similarity 0.86)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.86)
• Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.86)
• InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.85)
• SpatialLM: Training Large Language Models for Structured Indoor Modeling (NeurIPS 2025 · similarity 0.85)