Tools

Namespaces

Variants

Views

Actions

Search

Contents

Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries. Thanks for all your past and future contributions.

{{Note|This is an entry in the [[Nokia Original Effect Wiki Challenge 2014Q2]]}}

+

This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied. The article will also describe key sections of code that are requirements for creating your own version of this app.

This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied. The article will also describe key sections of code that are requirements for creating your own version of this app.

−

''Enter article metadata as described below. Note that this template can be placed anywhere in the article. Do not remove parameters that you do not use''

In order to use the MF Transform that was just created, there are two steps that must be taken.

+

# The C++ app must be referenced in the C# app. WP8.1 to WP8.1 and Win8 to Win8.

+

# The MF Transform must be registered.

+

+

In a desktop app, this would be done in the registry. In a WinRT app, this is done in the Manifest file in an XML editor. Right-click on the Package.appxmanifest file in the WP8.1 app, select "Open with...", and choose an XML editor. In between Applications and Capabilities, insert:

Revision as of 03:17, 8 June 2014

This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied. The article will also describe key sections of code that are requirements for creating your own version of this app.

Introduction

With the introduction of universal apps and WinRT for Windows Phone 8.1, the Silverlight/WP8.0 method of accessing the camera has changed. Within the latest Nokia Imaging SDK is a IImageProvider called CameraPreviewImageSource which allows you to process frames from a video stream before capturing a final image to process. However, unless you were to provide your own video encoder, you cannot record the video being output by the CameraPreviewImageSource.

WinRT provides a MediaCapture class which provides video preview, image capture, and video record capabilities. The preview can be output to a new XAML class called a CaptureElement. There is no traditional way to apply the Nokia Imaging SDK to the preview (or record) output of the MediaCapture class. However, the MediaCapture class will accept Media Foundation Transforms through the AddEffectAsync method. This article will describe the key sections of code that are required to add a Media Foundation Transform to the app and how to incorporate the Nokia Imaging SDk into it.

Many parts of this app were inspired by or even pulled directly from code provided by other developers on this site.

References coming soon....

C++ Universal Component

The Media Foundation Transform must be coded in C++ and the interfaces that must be implemented within the transform class are quite complicated. Thankfully, there exist several samples of a Media Foundation transform that do not require much modification for our purposes. A good sample to start with is the MediaExtensions sample from the Univeral Apps sample set from Microsoft. The sample contains a GrayscaleTransform component from which this app's transform is based.

Preparing the C++ Component

Add Media Foundation Libraries

Next, the Media Foundation libraries need to be added to the linker. For each C++ project (WP8.1 and Win8), right-click on the project name and select properties. Expand the Linker group and select Input. Change the Configuration (at the top) from Active to All Configurations and change Platform from Active to All Platforms so that the following changes are applied to every configuration and platform.

To the end of the "Additional Dependencies" block, add mfplat.lib and mfuuid.lib separated by a semi-colon(;).

Add Common Folder from sample & Include Directories

At the root of the sample app (this app or the MediaExtensions app) is a folder labeled Common which contains several C++ header files (AsyncCB.h, CritSec.h, ExtensionsDefs.h, etc) . These are necessary for the media transform code to compile. Copy the entire Common folder to the root of your own solution.

Next, go to the property page for each C++ project in your app and change the configuration and platform to all as before. Expand the C/C++ menu and click on General. To the end of the Addition Include Directories, add a semicolon (;) and the path "..\..\Common" so that VS2013 can find the header files we just copied.

Add the Nokia Imaging SDK

Add the Nokia Imaging SDK to all projects in the solution using NuGet. You do NOT have to manually add a reference to the Imaging SDK to the C++ projects. They will not appear as a reference under the properties, but they will be included in the project anyways.

Add NativeBuffer.h

The Transform Class

To the Shared C++ project Add both a .h header and .cpp c++ file. Creating these two files from scratch is beyond the scope of this article. Instead copy the contents of the ImagingEffect.cpp and ImagingEffect.h from the sample to your project. (These files are too large to post directly in this article.) There are several functions within this class that this article will highlight in the next section so that you can modify or add to this class.

Static Transform Functions for NV12 & YUY2

Most of the work of the transform happens within the non-class functions TransformImage_NV12 and TransformImage_YUY2. These functions transform the video frame at the pointer pSrc and copy the new, transformed image to the pointer pDest. Another important variable within these functions is the pointer to a wide string, filterParams. This string contains parameters from the C# project that define which filters to use and what parameters to use with them.

In order to use the Nokia Imaging SDK, the first step is to transform the raw byte data into a Bitmap. This is accomplished for the NV12 function using another function pulled directly from galazzo
's article again [[1]]. For YUY2 to Bitmap, the function was modified for the YUY2 format.

ApplyingImagingFilters function

The use of the Nokia Imaging SDK in a C++ app is similar to a C# app. First, a source must be created. Then, several Effects or Filters are applied. Finally, the result is rendered to a designated target. In this app, the source is a BitmapImageSource that uses the previously created Bitmap in its constructor.

Next, filters are added to a FilterEffect via a loop run using the filterParams parameter. This will be described in the next section.

Finally, a BitmapRenderer is created using a specified ColorMode so that the output matches the format of the input. In the Nokia Imaging SDK, NV12 matches to Yuv420Sp and YUY2 matches to Yuv422_Y1UY2V. A pointer to the raw byte data is created from the IBuffer internal to the newly created Bitmap. This byte data is then copied to the pointer, pDest.

Note: NV12 and YUY2 are two different formats for video that uses luminance and chroma data instead of straight RGBA data. These are the only two formats used by the Nokia Lumia 920 and the LifeCam HD-5000 webcam.

Parsing the Filter Parameters & Creating the Filters

In this app, the filter parameters are passed solely as text data. Since C++ does not have Reflection function like C# does, the strings have to be matched to class names manually. In the string parameter, filterParamters, filters are separated by semicolons (;) and the specific arguments are separated by comma (,). Within the GetFilter function, each filter name is matched up and the number of parameters is counted to determine which filter is returned to the main function

Warning: Currently, only LomoFilter and part of SolarizeFilter are ready to use with this solution. The template for using any other filter can be copied directly from these two.

CImagingEffect::SetProperties

This function allows the C++ transform to receive data from the C# portion of the app. When the transform is applied to the MediaCapture class via AddEffectAsync, only a string variable is passes and no instance of the transform is returned to the C# portion of the app. Therefore, one cannot pass data to the transform via conventional means. However, one of the parameters of the AddEffectAsync method takes an IPropertySet interface. This is essentially a dictionary of KeyValuePairs that must use WinRT-types.

This parameter comes through to the MF Transform through the SetProperties function.

This function parses the content of the IPropertySet and converts it to a std::wstring in the class called m_altFilterParams. However, the transform functions cannot access this class-member variable, so it must be passed via a pointer as an argument to the transform functions. This is done in OnProcessOutput:

C# Universal App

Preparing the main App

In order to use the MF Transform that was just created, there are two steps that must be taken.

The C++ app must be referenced in the C# app. WP8.1 to WP8.1 and Win8 to Win8.

The MF Transform must be registered.

In a desktop app, this would be done in the registry. In a WinRT app, this is done in the Manifest file in an XML editor. Right-click on the Package.appxmanifest file in the WP8.1 app, select "Open with...", and choose an XML editor. In between Applications and Capabilities, insert: