SpecTRe-GS: Modeling Highly Specular Surfaces with Reflected Nearby Objects by Tracing Rays in 3D Gaussian Splatting

3D Gaussian Splatting (3DGS), a recently emerged multi-view 3D reconstruction technique, has shown significant advantages in real-time rendering and explicit editing. However, 3DGS encounters challenges in the accurate modeling of both high-frequency view-dependent appearances and global illumination effects, including inter-reflection. This paper introduces SpecTRe-GS, which addresses these challenges and models highly Specular surfaces that reflect nearby objects through Tracing Rays in 3D Gaussian Splatting. SpecTRe-GS separately models reflections from highly specular and rough surfaces to leverage the distinctions between their reflective properties and integrates an efficient ray tracer within the 3DGS framework for querying secondary rays, thus achieving fast and accurate rendering. Also, it incorporates normal prior guidance and joint geometry optimization at various stages of the training process to enhance geometry reconstruction for undistorted reflections. Experiments on both synthetic and real-world scenes demonstrate the superiority of SpecTRe-GS compared to existing 3DGS-based methods in capturing highly specular inter-reflections and also showcase its editing applications.

Overview of SpecTRe-GS. The scene is represented as a Gaussian point cloud, where each Gaussian is associated with geometry and shading attributes used to render the depth map $D$, normal map $N$, diffuse reflection $I_{\rm diff}$, specular reflectance $A_{\rm spec}$, and rough surface reflection $I_{\rm rs}$ through splatting. Highly specular surface reflection $I_{\rm ss}$ is formulated as the sum of the diffuse reflection $I_{\rm diff}$ and the specular reflection computed from the incident radiance $I_{\rm i}$ and specular reflectance $A_{\rm spec}$. The incident radiance $I_{\rm i}$ is determined along reflected secondary rays given by the scene geometry $D$ and $N$, with the indirect component $I_{\rm ind}$ and visibility $V_{\rm i}$ evaluated via efficient ray tracing within the Gaussian point cloud, and the direct component $I_{\rm dir}$ modeled by an optimizable environment map. The scene geometry is enhanced using normal prior guidance, numerical gradients from ray-traced incident radiance, and other training signals to achieve undistorted reflections.