Abstract

Casually-taken portrait photographs often suffer from unflattering lightingand shadowing because of suboptimal conditions in the environment. Aestheticqualities such as the position and softness of shadows and the lighting ratiobetween the bright and dark parts of the face are frequently determined by theconstraints of the environment rather than by the photographer. Professionalsaddress this issue by adding light shaping tools such as scrims, bounce cards,and flashes. In this paper, we present a computational approach that givescasual photographers some of this control, thereby allowing poorly-litportraits to be relit post-capture in a realistic and easily-controllable way.Our approach relies on a pair of neural networks---one to remove foreignshadows cast by external objects, and another to soften facial shadows cast bythe features of the subject and to add a synthetic fill light to improve thelighting ratio. To train our first network we construct a dataset of real-worldportraits wherein synthetic foreign shadows are rendered onto the face, and weshow that our network learns to remove those unwanted shadows. To train oursecond network we use a dataset of Light Stage scans of human subjects toconstruct input/output pairs of input images harshly lit by a small lightsource, and variably softened and fill-lit output images of each face. Wepropose a way to explicitly encode facial symmetry and show that our datasetand training procedure enable the model to generalize to images taken in thewild. Together, these networks enable the realistic and aesthetically pleasingenhancement of shadows and lights in real-world portrait images