Style Transfer with StyleThief

Besir Kurtulmus

1 year ago

Style transfer is a term used for reimagining an image with the style of a given piece of art. Recently, various research groups have proposed different approaches to do style transfer. Generally speaking, there are trade-offs between these different techniques.

For example, in one of our previous spotlights we talked about DeepFilter, where you train a model based on a style, and stylize images almost instantaneously with that trained model. You would train for about a day, and later be able to stylize images rapidly. The biggest issue with this technique is that training wouldn’t always yield the best results. You would then need to train it multiple times, which could easily add up to a few days.

StyleThief works differently from DeepFilter. It takes a long time to train for every sample image, but is more robust and yields better stylized images. It is a trade-off between speed and quality.

How does it work?

The algorithm uses a Convolutional Neural Network to analyze the photo. It detects high-level features that roughly represents the overall shape of objects in the photo, then then calculates a feature space, which is essentially the correlation between the different features in different layers of the CNN.

At each iteration of processing, it re-applies these correlations to the images while keeping the high-level features such as objects. The end result is an image which takes on the applied style while keeping the original content (high-level features).

Pick a style image, and a content image. I’ve picked the The Great Wave of Kanagawa as a style image, and my own profile photo as the content image. You can always pick something else if you want.

Now, by running the code snippets (in Python) below, we’ll be able to stylize our image with the given style image! Note that, since this algorithm takes a bit longer to run, notice that we need changed our default running time from 5 minutes up to the maximum of 50 minutes, via “set_options(timeout=3000)“.

To find the stylized image, go to your Hosted Data at Algorithmia, and then look for the bkyan/StyleThief folder within the Algorithm data section. The output file will appear in that folder.

Conclusion

StyleThief is an algorithm that utilizes the power of deep learning to stylize your photos into cool, trippy, and fun photos. It uses a special technique called style transfer that looks for low-level features in the style image, and applies them to the high-level features (aka. Objects, building, humans etc.) in the original image.

The magic and heavy lifting is handled by the Algorithmia API, so you can just focus on developing your app without worrying about scalability issues.

Please let us know if you’ve used any of the filters in your app or service via @algorithmia on Twitter!

We would like to thank @bkyan for adding this algorithm to the marketplace.