Deep Image Blending
posted on: arXiv 2020.4
In this paper, they proposed two-stage blending algorithm, which first seamlessly blends the object onto the background, and then further refine the texture and style of the blending region. To apply Poisson blending, they presented a differentiable Poisson gradient loss to gain the equivalent efforts and easily combine with other losses. Their algorithm doesn’t rely on any training data and could generalize to any source and target images, such as real-world images and styled paintings. Experiments shows that their algorithm outperforms baselines including Poisson blending, style transfer and deep image harmonization.
Pros:
- It tries to solve the main problems faced in many blending/harmonization methods. For Poisson blending, the style and texture inside the foreground region is not consistent with the background. While in style transfer and image harmonization, the boundary consistency remains a problem. In this two-stage algorithm, it focuses on each problem in each stage and could generate visual pleasant results.
- The Poisson gradient loss they present is reasonable variant of Poisson blending, which could be easily applied in deep learning and combined with other losses.
- They also use the style loss and the content loss to maintain style consistency and maintain more content information. But different from other content similarity loss [1] or style loss, it’s applied on feature maps, making content close to foreground and style close to background.
- It uses pretrained VGG to extract feature maps used for style and content loss, and apply a Laplacian filter to compute gradient blending loss, so it doesn’t rely on any training data.
Cons:
The VGG is pretrained on ImageNet, I wonder whether the feature extracted from a styled painting is reasonable.
As there is no ground truth, it could only perform user studies. No quantity comparison between baselines.
[1] Unsupervised Pixel–Level Domain Adaptation with Generative Adversarial Networks