TransFill: Reference-guided Image Inpainting

by Merging Multiple Color and Spatial Transformations

CVPR 2021

Results of our reference-guided inpainting for user-provided images. We show multiple practical applications like replacing and removing foreground people and objects. Each triad shows the target image with the hole, the source image used as a reference, and the inpainting result. Our method has strong performance and addresses challenging real-world issues such as parallax, 90 degree image rotations, and lighting inconsistency between the source and target images. Some of the above images are from the dataset for the ACM SIGGRAPH 2016 paper Automatic Triage for a Photo Series, Huiwen Chang, Fisher Yu, Jue Wang, Douglas Ashley, Adam Finkelstein..

Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image. Most existing technologies exploit patch similarities within the image, or leverage large-scale training data to fill the hole using learned semantic and texture information. However, due to the ill-posed nature of the inpainting task, such methods struggle to complete larger holes containing complicated scenes. In this paper, we propose TransFill, a multi-homography transformed fusion method to fill the hole by referring to another source image that shares scene contents with the target image. We first align the source image to the target image by estimating multiple homographies guided by different depth levels. We then learn to adjust the color and apply a pixel-level warping to each homography-warped source image to make it more consistent with the target. Finally, a pixel-level fusion module is learned to selectively merge the different proposals. Our method achieves state-of-the-art performance on pairs of images across a variety of wide baselines and color differences, and generalizes to user-provided image pairs.

Intermediate Results Visualization
Unfolding the whole pipeline to visualize the intermediate results of each module.

Inpainting Results