Research map / Large language models / AI and society
NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Gaussian spatial cursor position map to the renderer context (ablation) | NeuralOS · Cursor position (730 eval examples) | x position error (pixels): 130.0 -> 1.6 | Improved |
| add joint training with the diffusion renderer after RNN pretraining (ablation) | NeuralOS · qualitative generation | frame sharpness | Improved |
| add scheduled sampling stage (ablation) | NeuralOS · long interaction sequences | pixel RMSE of generated frames | Improved |
| add RNN pretraining with MSE loss before joint training | NeuralOS · training | RNN gradient use | Improved |
| replace transformer encoder with hierarchical RNN (ablation) | NeuralOS · inference memory, 1024 s video | GPU memory (GB): >40 (OOM) -> 11.96 | Improved |
| add random exploration data to agent-generated demonstrations | NeuralOS · training data | spurious correlations | Improved |
| use recurrent state instead of a fixed 64-frame context (vs adapted DIAMOND) | NeuralOS · OS simulation, equal compute | pixel RMSE over generated steps | Improved |
| fine-tune DIAMOND with scheduled sampling (p=0.05) | adapted DIAMOND · OS simulation, equal compute | application-launch transitions | Mixed |
| extend test delay beyond the training delay (64, 128, 256 frames vs 8 in training) | NeuralOS · Folder presence after reopening | fraction predicted has folder: 0.52 -> 0.60 -> 0.62 | Mixed |
| crop bottom 40 pixels of videos in human evaluation (ablation) | NeuralOS · Human identification of real OS | human success rate (%): 58.3 -> 51.7 (10s) | Improved |
Closest papers by meaning
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.84)
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models (CVPR 2026 · similarity 0.83)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.83)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.82)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.82)
- OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
