Toward Realistic Image Compositing with Adversarial Learning
posted on: CVPR2019
Generating a realistic composite image requires both the adjustment of color and illumination and the geometrical transformations. In this paper, based on this observation, they proposed an image composition network consisting of four parts. The transformation network and refinement network act together as the generative compositing model. In this part, they apply spatial transformer network to predict the geometric correction function and linear brightness and contrast model to predict color correction parameters. The refinement network with encoder-decoder architecture is used to deal with boundary artifacts. Adversarial loss is applied to classify real images and generated composite images. They also add an additional segmentation network to predict the foreground mask. To avoid model from removing the foreground during geometric transformation, they propose a geometric loss to penalize large transformations and too small foreground masks.
They conducted experiments on synthesized 3-D images and COCO images. For synthesized 3-D images, they generate foreground, background and ground-truth composite images. For COCO, they apply a similar procedure to DIH, processing objects on real images to generate training images. The main difference is that, besides color distortion, they also use another auxiliary mask to simulate the boundary mismatch. They use only user studies to compare between baselines.
At the end, they also present to use image manipulation detection model RGB-N to detect different methods, demonstrating the realism of their generated composites.
Pros:
It a comprehensive method in image compositing field. Geometrical and color consistent adjustment is vital for realistic composites, which matches our intuition.
The experiment about image manipulation detection model is also a good way to compare different baselines in image harmonization.
Cons:
The purpose and effectiveness of additional segmentation network is not so clear. And the effectiveness of refinement network is not presented.
Though their model is constructed with many reasonable considerations, their experiments are too simple to test the overall effects.
They use only user studies to compare between baselines.