PlaneStereo: Plane-aware Multi-view Stereo

Learning-based multi-view stereo (MVS) algorithms have demonstrated great potential for depth estimation in recent years. However, they still struggle to estimate accurate depth in texture-less planar regions, which limits their reconstruction performance in man-made scenes. In this paper, we propose PlaneStereo, a new framework that utilizes planar prior to facilitate the depth estimation. Our key intuition is that pixels inside a plane share the same set of plane parameters, which can be estimated collectively using information inside the whole plane. Specifically, our method first segments planes in the reference image, and then fits 3D plane parameters for each segmented plane by solving a linear system using high-confidence depth predictions inside the plane. This allows us to recover the plane parameters accurately, which can be converted to accurate depth values for each point in the plane, improving the depth prediction for low-textured local regions. This process is fully differentiable and can be integrated into existing learning-based MVS algorithms. Experiments show that using our method consistently improves the performance of existing stereo matching and MVS algorithms on DeMoN and ScanNet datasets, achieving state-of-the-art performance.