PicFx – Windows Phone Picture Effects Application – Part 2

The first part of this short series showed how to create the base Windows Phone application and how to implement a Black & White and Sepia effect. The basic Windows Phone picture manipulation workflow was explained, and I showed how to load, resize, take, and save an image. The User Interface with the Pivot control template was introduced and some important Windows Phone development key points were also discussed. We also learned how to implement the Black & White and the Sepia effect with the reusable Tint and Contrast & Brightness modification effects.

In this second, final part of the series, we will learn how to make the application more responsive by offloading the image processing computation to a background thread. Furthermore, how to implement a nice vintage Polaroid-like and a miniature faking (tilt shift) effect will be demonstrated, along with how to brand the finished image with a custom logo.

The App in Action

The video below introduces the complete PicFx application features and demonstrates how to use them. It was recorded with the application running in the emulator.

How it works

In the last article, we drilled down from the UI Crust with the Pivot control template and the Windows Phone Application Bar through the UI Mantle with the UI functionality until we finally reached the Effects Core with the image processing algorithms.

The User Interface

As in the first part, we again start our journey on the surface of the application.

An IndeterminateProgressBar was added to the first Pivot item overlaying the Image, which shows the selected picture. The ProgressBar is hidden by default and only made visible when the picture with the full resolution is processed and saved.

The ListBox with the thumbnails of the effects (see Figure 1) is still data-bound to the StaticResource “effects,” an instance of the EffectItems class that consists of EffectItem elements:

The Code Behind

The first part introduced the Windows Phone picture manipulation workflow and explained how to load, resize, take, and save an image. This section will show how to keep the UI responsive by performing the image processing asynchronously.

The two new effects introduced in this article are more computationally expensive. If these would be applied to the original sized picture, the UI thread would get blocked for a few seconds. This is a No Go for a professional application and in order to pass the Marketplace validation, an app has to be responsive and needs to avoid hang-ups. This and other important requirements are defined in the official Windows Phone 7 Application Certification Requirements document.

Asynchronous Processing

To achieve a good responsiveness of the application, the work has to be offloaded from the UI thread to a background thread. Here is where Silverlight's multi-threading strength comes into play.

There is only one problem—due to its base classes, the WriteableBitmap can't be used in a non-UI thread. As we know from the first article, the WriteableBitmap uses the RGB color space to represent the pixels. It's actually just a 32-bit integer array that stores the alpha, red, green, and blue (ARGB) byte components for all the pixels in a 1D array, and stores the width and the height as integer properties. This leads us to the natural solution: perform the whole cascade of image processing effects with an integer array (pixels) along with the width and height and only copy the final result back to WriteableBitmap on the UI thread.

No sooner said than done, the following code is executed when the user hits the Save button:

The ProgressBar is shown during the asynchronous processing and the Opacity of the Image control is reduced (Figure 2). The actual image processing with the selected effect is performed on a background thread. Using the ThreadPool class and its QueueUserWorkItem method accomplishes this. The ThreadPool provides a pool of threads and the QueueUserWorkItem is used to queue a work operation for processing. The main benefit is that resources aren't unnecessarily hogged—the creation of a thread takes some time and each thread needs certain resources such as its own memory stack. Also note that, for common computational scenarios, it's best to keep a balance between threads and processor cores. A thread pool avoids the creation overhead through a certain amount of threads that are kept alive and all the queued work is executed one after another by these threads.

The IEffect's new Process method overload with the pixels integer array and size parameters is used inside the background thread processing. Read more about the IEffect interface change below.

After the effects processing chain is done, a new WriteableBitmap is then instantiated on the UI thread with the use of the Page's Dispatcher, which executes code on the UI thread. Then the integer pixels array is copied to the WriteableBitmap's Pixels property with the fast BlockCopy. The BlockCopy method copies a block of bytes in one single operation in memory, just like the good ol' memcpy.

The final bitmap is then saved to the picture library/photo album with the SaveToMediaLibrary extension method that was introduced in the first part. Finally, the ProgessBar is again hidden and the Image's opacity is restored.

Figure 2: Saving a picure shows the ProgressBar over the semi-transparent Image

The Effects

Now that we made the image processing asynchronous and the UI is responsive even when complex computations are performed, it's time to leverage this feature for some advanced effects.

The IEffect interface had to be changed for the asynchronous WriteableBitmap-less processing. The new Process method overload expects the pixels as ARGB32 integer array, and the width and height of the bitmap as parameters. The return value is the processed bitmap as integer pixel array of the same size.

Figure 3: The changed IEffect interface

As we learned in the first part, the end user effects are called composite effects (outer core), which are made out of various, reusable base effects (inner core).

The Vintage Polaroid-like Effect

This composite effect gives a picture an old-touch so it looks like it was taken with an old, Polaroid-like camera.

Figure 4: Vintage Polaroid-like effect applied to the sample picture

The Polaroid-like composite effect uses three internal base effects: a Gaussian blur, an effect that adds a black vignette, and the Tint base effects introduced in the first part (Figure 5).

Figure 5: The class diagram of the PolaroidEffect

Old photos weren't as sharp as modern photos and therefore the Blurriness property defines how much the picture will be blurred. The Vignette property controls the size of the round vignette. Additionally, the amount of tinting and the tint color can be changed with properties.

First the input is blurred and the vignette is added. Afterward, a new Sepia-tinted version of the processed image is created. In the last processing step, the non-tinted and the tinted bitmap are mixed together with the use of the new BitmapMixer class. This results in a slight Sepia tint rather than the full Sepia tone.

The New Base Effects

The vintage Polaroid-like effect uses three base effects and a mixer to achieve the old look. Now it's time to see how its Inner Core effects work.

The Blur Effect

There are several different available blur algorithms, and one of the most common is the Gaussian blur. The Gaussian blur applies a Gaussian function to an image in order to smooth it and reduce details. The naïve implementation uses a convolution kernel, which is basically a 2D array of n x n elements. In the case of a Gaussian filter the kernel values represent a discrete 2D Gaussian function, which has the typical bell shape. The usual kernel has a size of 5 x 5, though other kernels range from 7 x 7 on up.

Figure 6: Gaussian blur applied to the sample image

During the actual image processing, a pixel of the input image is multiplied with all of the kernel elements. The products of the pixel-kernel-element multiplication are summed and divided, and then the result is assigned as the output pixel. This is done for all the pixels of the input image.

The disadvantage of this approach is that the computation time increases when the blurring is increased. In order to blur the image more, the kernel size is usually enlarged, thus meaning n x n multiplications need to be performed for each pixel. Fortunately, in 1995 Ian T. Young and Lucas J. van Vliet invented a better algorithm that is independent of the width. They describe the method in detail in their paper, “Recursive implementation of the Gaussian filter.”

It gets even better—Andrew Marshall already implemented the recursive Gaussian filter in C# for his Silverlight and CUDA interop blog post and allowed me the use it. As you can imagine, the implementation is quite complex and could make up an article on its own. In fact, Young & van Vliet already wrote this article by writing their paper. Please read it if you want to know the mathematical details behind the GaussianBlurEffect class.

The Vignette Effect

The effect of vignetting reduces the brightness of the pixels towards the edges. This is done by computing the pixel's distance to the center and multiplying this with the pixel color. This generates the vignette effect as a fadeout to black towards the edges.

The x and y coordinate of the image's center and the aspect ratio are calculated. As you can see, only fast integer operations are used again.

Inside the loop, the color components of each input pixel are extracted and the distance vector to the center is calculated with respect to the picture's aspect ratio. The squared length of the distance vector is tested against the minimum vignette size. If the pixel falls within the range, the scaled distance length is multiplied with each color component. The result is an adapted brightness as described above. The last step ensures that the color components are in the byte range and then combines these to the result integer pixel color.

The BitmapMixer

As the name might imply, the purpose of the BitmapMixer class is to mix two images. The Mix method mixes two ARGB32 integer bitmaps of the same size and returns the mixed bitmap. This is actually an alpha blending operation where the Mixture property defines the opacity of the input2 image. A Mixture value of 0 means input1 is fully visible and a value of 1 means that input2 is shown—everything in between is a mix of both.

The color components of the two input images are extracted. Each color component of input2 is then multiplied with the Mixture factor and input2 is multiplied with the inverse of the Mixture. Then, both color component products are added and the new values are combined to form the new integer output pixel.

The Tilt Shift Effect

Now that we've learned the details of some of the new Inner Core effects, it's time to use some of them in a different combination and make an interesting Outer Core effect.

The digital tilt shift effect lets a scene look like a miniature scale model. It's quite popular nowadays and you might have seen it applied to video in some ads. It's commonly called miniature faking and produces a nice result if it's applied to a photo that was taken from a high angle.

Figure 8: The tilt shift Effect applied to a photo of Dresden that I have taken from a Ferris wheel

In the first processing stage the TiltShiftEffect increases the contrast of the image with the BrightnessContrastModification effect, which was introduced in the first part. Afterward, the picture gets blurred with the GaussianBlurEffect. The blurred version is then combined with the non-blurred to produce the shallow depth of field of a close-up shot.

Figure 9: The class diagram of the TiltShiftEffect

The UpperFallOff property defines the relative y coordinate where the depth of field (camera focus) is completely faded out. The LowerFadeOff defines the lower focus counterpart:

As you can see, the processing is split into three methods and two member variables are used to cache both the contrast-increased result and the blurred result. This is useful when only the FallOff properties are changed interactively in real-time, which is described below.

Figure 10: The simulated depth of field fade out

The actual processing mixes the contrast-increased image and the blurred image by using a linear fading function. Figure 10 illustrates this. The red color represents the blurred version and the gray stands for the contrast-increased image.

This fading uses the FallOff properties, converts these properties into absolute values, and calculates some y coordinates, which are needed for the fade in/out. Inside the loop, the color components of both bitmaps are extracted and the mixture factor is computed. Then the color components are multiplied with the factors like in the BitmapMixer's Mix method. The last step combines the component results and sets the integer pixel of the result image.

Multitouch Focus Fade Out Manipulation

The Windows Phone is a nice multitouch device with very good usability. Its multitouch power is used in the PicFx app to let the user interactively change the FallOff properties of the TiltShiftEffect; therefore, the focused area can be altered in an intuitive way.

Silverlight and the Windows Phone Silverlight version provide the static Touch class, which has only one member, the FrameReported event. This event is fired each time a set of touch points is registered.

An event handler is attached in the Initialize method of the MainPage. Please note that one would actually encapsulate the following code in a separate class like ViewModel for the effect, but I decided to leave this out to keep the code simpler and focused on the nitty gritty.

Every time the multitouch event is fired, the SetTiltShiftFocus method gets called. This method converts the absolute coordinates into relative and assigns the values to the appropriate properties. The topmost point is always interpreted as UpperFallOff.

Two small, gray rectangles are drawn at the position of the FallOff values to give the user some feedback. This is done with the WriteableBitmapEx' FillRectangle extension method. To keep the UI responsive, the ProcessOnlyFocusFadeOff method of the TiltShiftEffect is called, and this method uses the cached contrast-increased and blurred images by mixing them. This speeds the process up a lot up.

The Watermark

Now it's time to brand our final image with a custom logo before it gets saved. This watermark is useful to customize or add information to a photo.

Figure 11: Watermark logo applied to the sample image

The Watermarker class has the Watermark property, which represents a WriteableBitmap that is used as watermark logo. The RelativeSize defines the size of the logo relative to the size of the input bitmap it should get applied to:

The constructor provides an easy way to pass a bitmap from the resource stream. In the Apply method, the watermark bitmap is scaled with the use of the WriteableBitmapEx' Resize method. After this the position is calculated, the watermark logo is blitted into the bottom right corner of the input image and the result is returned. Note that the WriteableBitmapEx' Blit method is used here.

An instance of the Watermarker class is created in the MainPage.xaml.cs.

The watermark is applied after the image processing was performed and before the picture gets saved to the media library.

Conclusion

In the first part we drilled down from the UI Crust with the Pivot control template and the Windows Phone Application Bar through the UI Mantle. Finally we reached the Effects Core with the Black & White, Sepia, BrightnessContrast, and Tint effects.

In this second part, we again journeyed to the core, starting on the surface in order to learn how to keep the UI responsive with asynchronous processing. We then entered the core and l explained the Polaroid-like vintage, its Gaussian blur, the Vignette effects, and the BitmapMixer. I also demonstrated the miniature faking Tilt Shift effect, including the multitouch manipulation of its parameters. The last step showed how to add a custom logo watermark to the final picture.

This short series, or two articles, if you will, has come to end. Yep, it's over now—BUT Coding4Fun has released this development stage of the PicFx app for free on the Marketplace! Furthermore, I continued my work on this project and shipped it with enhanced effects, without watermark, but with extra features and a bunch of new effects, including essential ones like auto adjust, soften and many more. Check out the app called Pictures Lab aimed to be nothing less than THE image effects addition to the Windows Phone Pictures Hub.

About The Author

René Schulte is a .Net, Silverlight and Windows Phone developer and Microsoft Silverlight MVP passionate about real-time computer graphics, physics, AI, and algorithms. He loves C#, Shaders, Augmented Reality, and computer vision. He started the SLARToolkit, the WriteableBitmapEx, and the Matrix3DEx Silverlight open source projects, and he has a Silverlight website powered by real time soft body physics. He is also a regular author for Microsoft's Coding4Fun. Contact information can be found on his Silverlight website, his blog, or via Twitter.

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation,
please create a new thread in our Forums, or
Contact Us and let us know.