Improvement and measurement of neural style transfer

Style transfer methods produce a transferred image which is a rendering of a content image in the manner of a style image. There is a rich literature of variant methods. We seek to understand how to improve style transfer: in particular, there is some evidence that cross-layer losses are helpful, and some evidence that optimization problems might present difficulties. To do so requires quantitative evaluation procedures, but current evaluation is qualitative, mostly involving user studies. We describe a novel quantitative evaluation procedure. Our procedure relies on two statistics: the Effectiveness (E) statistic measures the extent that a given style has been transferred to the target, and the Coherence (C) statistic measures the extent to which the original image's content is preserved. Our statistics are calibrated to human preference: targets with larger values of E (resp C) will reliably be preferred by human subjects in comparisons of style (resp. content). We use these statistics to investigate relative performance of a number of recent style transfer methods, revealing a number of intriguing properties. {Our experiments pool multiple style transfers from many different styles to many different content images using many different style weights, allowing us to make general statements about what influences style transfer. }Admissible methods lie on a Pareto frontier (i.e. improving E reduces C, or vice versa). Three methods are admissible: Universal style transfer produces very good C but weak E; modifying the optimization used for Gatys' loss produces a method with strong E and strong C; and a modified cross-layer method has slightly better E at strong cost in C. While the histogram loss improves the E statistics of Gatys' method, it does not make the method admissible. Surprisingly, style weights have relatively little effect, and most variability in transfer is explained by the style itself (meaning experimenters can be misguided by selecting styles).

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2019

Author dc:creator

• Yeh, Mao-Chuang

Contributors dc:contributor

• Forsyth, David A.

• style transfer
• gram matrix
• texture synthesis

Statement dc:rights

• Copyright 2018 Mao-Chuang Yeh

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/102527

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/102527