NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects

ICCV 2023; ICCV23 TRICKY workshop Spotlight
School of Computer and Communication Sciences, EPFL, Switzerland
teaser image

NEMTO is capable of high-quality novel view synthesis and relighting through optimizing an end-to-end neural representation for a transparent object.


We propose NEMTO, the first end-to-end neural rendering pipeline to model 3D transparent objects with complex geometry and unknown indices of refraction. Commonly used appearance modeling such as the Disney BSDF model cannot accurately address this challenging problem due to the complex light paths bending through refractions and the strong dependency of surface appearance on illumination. With 2D images of the transparent object as input, our method is capable of high-quality novel view and relighting synthesis. We leverage implicit Signed Distance Functions (SDF) to model the object geometry and propose a refraction-aware ray bending network to model the effects of light refraction within the object. Our ray bending network is more tolerant to geometric inaccuracies than traditional physically-based methods for rendering transparent objects. We provide extensive evaluations on both synthetic and real-world datasets to demonstrate our high-quality synthesis and the applicability of our method.


Overview of our model architecture.

model arch

Novel View Synthesis Comparison

Comparsion with baseline methods on novel view synthesis.

novel view

Real world experiments

We demonstrate the effectiveness of our method on novel view synthesis for real-world transparent objects. With NEMTO, we can relight captured transparent objects with unseen environment maps.

novel view



          author={Wang, Dongqing and Zhang, Tong and S{\"u}sstrunk, Sabine},
          title     = {{NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects}},
          booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
          month     = {October},
          year      = {2023}}