A deep-learning tool that lets you clone an artistic style onto a photo

May 10, 2017

“Deep Photo Style Transfer” is a cool new artificial-intelligence image-editing software tool that lets you transfer a style from another (“reference”) photo onto your own photo, as shown in the above examples.

An open-access arXiv paper by Cornell University computer scientists and Adobe collaborators explains that the tool can transpose the look of one photo (such as the time of day, weather, season, and artistic effects) onto your photo, making it reminiscent of a painting, but that is still photorealistic.

The algorithm also handles extreme mismatch of forms, such as transferring a fireball to a perfume bottle. (credit: Fujun Luan et al.)

“What motivated us is the idea that style could be imprinted on a photograph, but it is still intrinsically the same photo, said Cornell computer science professor Kavita Bala. “This turned out to be incredibly hard. The key insight finally was about preserving boundaries and edges while still transferring the style.”

To do that, the researchers created deep-learning software that can add a neural network layer that pays close attention to edges within the image, like the border between a tree and a lake.

This research is supported by a Google Faculty Re-search Award and NSF awards.

Abstract of Deep Photo Style Transfer

This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits.

comments 7

This article brings back fond memories of the early days of Mac” I had a great designer in Dublin and we were producing visual business cards doing just this kind of montage,we were getting orders bigtime,we couldn’t get a printer to print them so we had to refill the cartridges by hand and cut the cards with a guillotine paper cutter,it was a new idea and the orders were loading up until i got a client who offered a hundred grand business business proposal,the designer got nervous and pulled the plug on the concept,that was the end of that pipe dream” the sales pitch went like this(you can have 10 Ferrari’s or a hundred Volkswagen’s.
Cheers.

Auto-coloring… I could sooooooo use this for animations. For way too long we’ve been deprived from being able to make full animations from just a few sketches and that’s one of the big things I think machine learning will be able to change in the next few years ^_^.

For many years I have felt that almost any artistic style by renown painters such as Van Gogh, Monet, Picasso etc where their style is easily recognizable, could be transferred to almost any picture made with a camera.
The trick is for the machine to learn the particular distinctive style of a painter and then translate it to a new different image. It would then be interesting if the machine guided a robot to paint the new image.
The reverse should also theoretically be possible, to take a painting of a great artist with distinctive style and turn it into a realistic image.
We can by analogy take this process to sound, especially one’s voice. You hypothetically could take the sound of any voice and real time convert someone else’s voice to the first voice sound. For example I could speak and it would come out in Winston Churchill’s voice etc. I could sing (badly) and it would sound like Pavarotti. You likewise could speak in any accent desired for any language.
It might be very beautiful to see a movie picture converted to the style of Van Gogh etc.
Although these ideas are pure fantasy now, it could be used to produce flawless voices for people who cannot speak or who have lost there vocal chords.

Where will it all end, real-time styling glasses that are the ultimate form of rose coloured glasses? People may stop improving the appearance of their environments as they can just live in denial of what it really looks like. Or perhaps our environmental design will move toward a more neutral form that maximises the quality of restyling devices?

People will have settings to recognise individuals and morph them in very specific ways that could be either complimentary, or very derogatory, and what ethical framework will we be able to apply to such private visions?

I discussed this using the term “procedural art” in the late ’80′s, paraphrasing:
You will be able to tell the computer, “give me a beach scene, no humans, evening, red sky, some clouds, large waves breaking to the left… No, too much red” And the computer would know your style from neural net experience. But that should lend itself to multiple levels of style and the emergence from that of whole new genres and conceptions of art, including immersion collectives, that could themselves create a whole new kind of AI.
I had a close friend in college, majoring in psych, who told me that he had concluded that the basis of life’s meaning was to live artistically. I didn’t get it at the time, but I may live long enough yet… Let’s see now; if art is an abstraction of the essence of things, and life is a regenerative process that depends upon implicate order…
DSM, have you read Vinge’s “Rainbows End?” The major underlying plot revolves around competing schools of art as experienced and created via collaborative artists in AR.