Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections

Dongbin Zhang1*, Chuming Wang1*;, Weitao Wang1, Peihao Li1, Minghan Qin1, Haoqian Wang1† (* indicates equal contribution, † means corresponding author)
1Tsinghua Shenzhen International Graduate School, Tsinghua University
Paper arXiv Code

We utilize 3D Gaussian Splatting with introduced separated intrinsic and dynamic appearance to reconstruct scenes from uncontrolled images, achieving high-quality results and a 1000 × rendering speed increase.

Abstract

Novel view synthesis from unconstrained in-the-wild images remains a meaningful but challenging task. The photometric variation and transient occluders in those unconstrained images make it difficult to reconstruct the original scene accurately. Previous approaches tackle the problem by introducing a global appearance feature in Neural Radiance Fields (NeRF). However, in the real world, the unique appearance of each tiny point in a scene is determined by its independent intrinsic material attributes and the varying environmental impacts it receives. Inspired by this fact, we propose Gaussian in the wild (GS-W), a method that uses 3D Gaussian points to reconstruct the scene and introduces separated intrinsic and dynamic appearance feature for each point, capturing the unchanged scene appearance along with dynamic variation like illumination and weather. Additionally, an adaptive sampling strategy is presented to allow each Gaussian point to focus on the local and detailed information more effectively. We also reduce the impact of transient occluders using a 2D visibility map. More experiments have demonstrated better reconstruction quality and details of GS-W compared to previous methods, with a 1000 × increase in rendering speed.

With an unconstrained image collection input, our method can render novel views with appearance tuning, achieving a significantly faster rendering speed 1000 × while maintaining state-of-the-art quality.

Reconstruction scenes with variant appearance

Novel View Synthesis

Appearance tuning

Images are rendered at the same camera pose with increasing weight of features extracted from the image. Our method incorporates environmental factors, like highlights on pillars and enhancing illumination, in a manner that is closer to human understanding.

Visual comparison on test set

Qualitative experimental results on three unconstrained datasets. GS-W recovers finer details of appearance and reconstructs more detailed geometry.

BibTeX

@article{zhang2024gaussian,
  title={Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections},
  author={Zhang, Dongbin and Wang, Chuming and Wang, Weitao and Li, Peihao and Qin, Minghan and Wang, Haoqian},
  journal={arXiv preprint arXiv:2403.15704},
  year={2024}
}