A Recipe for Generating 3D Worlds From a Single Image

We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page:this https URL

@article{schwarz2025_2503.16611,
  title={ A Recipe for Generating 3D Worlds From a Single Image },
  author={ Katja Schwarz and Denys Rozumnyi and Samuel Rota Bulò and Lorenzo Porzi and Peter Kontschieder },
  journal={arXiv preprint arXiv:2503.16611},
  year={ 2025 }
}