Research map / Large language models / Language, safety and interpretability
SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SysCaps, LLM-synthesized text descriptions of simulator metadata, and a lightweight multimodal text-plus-timeseries surrogate that uses them to predict building energy and wind farm power.
Problem
Surrogate models for complex energy systems must generalize across system configurations given tabular attributes, and it is unclear whether text (especially natural language) interfaces are suitable for real-world tabular timeseries regression.
Approach
Encode system attributes as text (key-value or LLM-written natural language "system captions"), embed them with a fine-tuned BERT/DistilBERT, broadcast the embedding over time, and fuse with weather timeseries via a bidirectional sequence encoder (LSTM or S4 SSM) to regress simulator outputs.
Key results
- SysCaps-kv with BERT and LSTM reached building-hourly NRMSE 0.439 vs 0.679 for LightGBM and 0.450 for one-hot SSM; SSM+BERT+SysCaps-kv reached stock-annual NRMSE 0.020 vs 0.094 for LightGBM.
- On wind farm wake data, SysCaps-nl with prompt augmentation reached NRMSE 0.035 vs 0.196 for LightGBM and 0.212 for one-hot ResNet.
- Synthetic caption attribute accuracy was 88.9-90.9% across short, medium, and long caption types.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace one-hot attribute encoding with SysCaps-kv using BERT | Building SSM surrogate · Buildings-900K held-out buildings | Stock-Annual NRMSE: 0.041 -> 0.020 | Helped |
| Use natural-language SysCaps instead of one-hot attributes | Building SSM surrogate · Buildings-900K held-out buildings | Stock-Annual NRMSE: 0.041 -> 0.035 | Helped |
| Use sequential LSTM or SSM encoding rather than ResNet or LightGBM | Building surrogate · Buildings-900K held-out buildings | Buildings-Hourly NRMSE: 0.634 -> 0.439 | Helped |
| Add system attributes to the SSM (ablation) | Building SSM surrogate · Buildings-900K held-out buildings | Buildings-Hourly NRMSE: 1.712 -> 0.450 | Helped |
| Fine-tune BERT for multimodal alignment (ablation) | Building SysCaps-nl surrogate · Buildings-900K held-out buildings | Stock-Annual NRMSE: 0.035 -> 0.356 | Helped |
| Use BERT instead of DistilBERT | Building SSM with SysCaps-nl · Buildings-900K held-out buildings | Stock-Annual NRMSE: 0.069 -> 0.035 | Helped |
| Use prompt augmentation to generate diverse natural-language captions (ablation) | Wind farm SysCaps-nl ResNet · Wind Farm Wake Modeling Dataset | NRMSE: 0.038 -> 0.035 | Helped |
| Replace one-hot attributes with SysCaps-nl | Wind farm ResNet surrogate · Wind Farm Wake Modeling Dataset | NRMSE: 0.212 -> 0.035 | Helped |
| Test with long captions after training on medium captions (ablation) | Building SSM · Buildings-900K held-out buildings | NRMSE: 0.53 -> 0.64 | Hurt |
| Replace building type with a semantic synonym in the caption | Natural-language SysCaps surrogate · Buildings-900K held-out buildings | NRMSE difference from original caption: 11/14 building type synonyms ... less than 13%; ... baselines ... 54% and 90% | Helped |
Surprising findings
- Fine-tuning BERT was necessary: an SysCaps-nl model without BERT fine-tuning had stock-annual NRMSE 0.356, far worse than fine-tuned variants.
- A model trained without any attribute information failed to learn the building task (building-hourly NRMSE 1.712).
Limitations
- Caption generation by llama-2-7b-chat introduces errors (missing or incorrect attributes, ~9-12%) and logical mistakes; numbers with commas hurt accuracy.
- BERT-style tokenizers interpolate poorly to unseen numerical values, and tail values (e.g., >100K sq ft) fail.
- No user studies with non-experts were conducted; approach only tested on two simulators with up to ~13-attribute systems.
Experimental setup
Datasets: Buildings-900K End-Use Load Profiles database Wind Farm Wake Modeling Dataset (FLORIS)
Benchmarks: Buildings-900K held-out buildings (NRMSE) Wind Farm Wake Modeling Dataset (NRMSE)
Metrics: NRMSE building-hourly NRMSE stock-annual NRMSE caption attribute classification accuracy
Baselines: LightGBM one-hot ResNet one-hot LSTM one-hot SSM SysCaps-kv
Builds on: BERT DistilBERT S4 LightGBM Time-LLM
Closest papers by meaning
- G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration (ICML 2025 · similarity 0.83)
- Interventionally Consistent Surrogates for Complex Simulation Models (NeurIPS 2024 · similarity 0.82)
- SMRS: advocating a unified reporting standard for surrogate models in the artificial intelligence era. (NeurIPS 2025 · similarity 0.82)
- NeuralBES: A Differentiable, Control-Aware Emulator for Scalable Building Energy Modeling (NeurIPS 2026 · similarity 0.82)
- BRIDGE: Bootstrapping Text to Control Time-Series Generation via Multi-Agent Iterative Optimization and Diffusion Modeling (ICML 2025 · similarity 0.82)
- EvoMesh: Adaptive Physical Simulation with Hierarchical Graph Evolutions (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.