Deep Painterly Harmonization
posted on: Computer Graphics Forum 2018
For local painting transfer, either photographic compositing or global painting transfer performs poorly. In this paper, they proposed a two-pass algorithm to transfer the style of a painting to the photo object pasted on it. The first pass aims for coarse harmonization by first performing a rough match of the color and texture properties. They designed a robust algorithm to deal with different painting styles. The second pass starts from the intermediate result in the first pass and focuses on removing spatial outliers to improve visual quality. More specifically, they adopt pretrained VGG to perform the transfers. In the first pass, they treat each layer independently during input-to-style mapping. The coarsely harmonized results are robust because a poor match in a layer can be compensated for by better matches in the other layers. In the second pass, they enforce consistency across layers and in image space during input-to-style mapping and make some constraints to the style loss. They used style and content loss, histogram loss and total variation loss following previous works. Given a foreground mask, the losses are computed and backpropagated only within the mask. As for the level of stylization varies in different styles of painting, the weights between style and histogram weights need to be considered. So they proposed to train a painting detector to predict the weights and assign to different styles of paintings.
Based on the output from the second pass, they also performed a two stage post-processing to deal with medium scale and large scale paintings. The first stage, chrominance denoising, suppresses the highest-frequency color artifacts after converting into CIE-lab color space. The second stage, patch synthesis, filters the base layer and averages the filtered base layer and the detail layer, which contains the high-frequency details.
Pros:
- Style transfer are mostly performed on the whole image, whether between photos or paintings. And transferring such global statistics to local area could lead to artifacts due to some irrelevant regions. They design an effective local approach to perform transfer on local region directly.
Cons:
- The two pass and the painting detector seems reasonable. But with the output from the second pass, they perform a two-stage postprocessing again, making it doubtful whether the second pass makes sense and how much the quality relies on the postprocessing.