RayNet: Learning Volumetric 3D Reconstruction

Recent methods based on Convolutional Neural Networks (CNN) allow learning the 3D Reconstruction task directly from data. However, they do not incorporate the physics of the image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. Our model integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.

The purpose of this site is to host the documentation page of our code that accompanies our CVPR 2018 paper with title RayNet:Learning Volumetric 3D Reconstruction with Ray Potentials.

Below you can find our:

• Supplementary material
• Poster from CVPR 2018
• Spotlight talk from CVPR 2018

Code Documentation

While this library was originally developed to accompany our CVPR publication, we additionally provide various console applications that can be easily used to perform 3D reconstruction using a set of images taken from known camera poses without the overhead of writing additional code.

Citation

If you use our library please also cite our paper. The bibtex can be found below:

@InProceedings{Paschalidou_2018_CVPR,
    author = {
        Paschalidou, Despoina and
        Ulusoy, Osman and
        Schmitt, Carolin and
        Van Gool, Luc and Geiger, Andreas
    },
    title = {RayNet: Learning Volumetric 3D Reconstruction With Ray Potentials},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2018}
}

License

raynet-mvs is released under the MIT license which practically allows anyone to do anything with it.