[GUIDE FOR DEVELOPERS] How to create HOMEBREW apps with NATIVE code on MANGO

When we were back on NoDo there were quite a few homebrew apps that used native code to apply tweaks to WP7 devices. Most of those apps seized to work after the device is upgraded to Mango. There a several reasons for this behavior. I've done research on this, because I wanted to make WP7 Root Tools compatible with Mango. In this topic I'd like to explain how developers can fix their apps to work on Mango again. It has taken me quite some time to compile this guide, but I hope to give the Homebrew development on WP7.5 Mango a boost.

This guide is NOT about creating homebrew executables (exe-files) for WP7. This guide aims to utilize native code DLL's (C++ / ARM) from within your Silverlight app.

Note that with native code you get access to a lot of extra API's. But that does not mean you automatically get access to resources you normally won't have access to. For example, you can use the CopyFile() API. But if you try to copy a file to the \Windows folder, you will get errorcode 0x4ec (1260), which means "Blocked by policy". So you are still bound to the rules of the sandbox of your app. If you want Full Root Access for your app, you have to wait for a new version of WP7 Root Tools, which will allow you to give your app root-access. I'm also working on an SDK for that, which wraps all common task into a neat managed library. But don't hold your breath for that, because it's all taking a bit longer than I expected.

To understand everything in this guide you need basic knowledge of C++, COM-interop and Silverlight for Windows Phone. If you are new to all this, you might want to do some reading on these topics first. Currently there is no way to debug the native code. The only thing you can do is create test-functions which return formatted debug-info. This makes things pretty difficult. Read the guide carefully, because a little mistake can make your app crash easily!

Important note: If you have any long-running tasks, they may work fine while you are debugging. But you need to make sure that you start a new thread to run this code. Because, when you run without debugger the WatchDog will monitor your application and if the User Interface thread is blocked for more than 10 seconds the WatchDog will exit your app ungracefully!

It has been suggested that native homebrew DLL's need to be signed with approved code-signing keys. This is in fact not true! You can use native DLL's on Mango devices, which are not signed at all!

Basically there are two reasons why homebrew apps are not working anymore:
- Interop Lock
- DLL's were built against libraries, which are not supported anymore on Mango

Interop Lock is discussed in this thread. Interop Lock is a new protection mechanism in WP7.5 Mango. Basically it means you can't use apps with ID_CAP_INTEROPSERVICES, unless a device is Interop Unlocked. Without ID_CAP_INTEROPSERVICES an app can't call any drivers. And most homebrew apps call these drivers directly or indirectly. So if an app uses the Interop Capability, it can only run on devices that are Interop Unlocked. If you're going to build an app that uses this capability on Mango, you'll have to give your users instructions on how to apply Interop Unlock on their device.

Most of the native code libraries that were used on NoDo, were based on a hand full of projects. These projects were created and then extended for their own needs by other developers. The result was that most of these projects had the same project-types and library-references. In Mango, a lot of DLL's that were not used anymore by Microsoft, have been removed from the OS. Mostly in the ShellCore. The DLL's were meant for MFC-type functionality, which was never even supported on WP7. Actually, these DLL's are not even used by the homebrew apps either, but there are references to these DLL's in the homebrew libraries, which will cause the library to fail loading into memory. You can see this behavior when you try to run an app with non-Mango-compatible native code on an Interop Unlocked device from within the Visual Studio 2010 development environment. When the COM-class is instantiated it will throw an COMException: "COM object with CLSID '{...}' cannot be created due to the following error: The request is not supported." This is errorcode 0x80070032. This exception is actually caused due to the fact that the previous call to RegisterComDll() failed. If you get the returnvalue of that function you should have 0. In this case the return-value is probably 0x8007007E, which is "Module Not Found". This actually means that you directly or indirectly refer to a DLL, which cannot be found on the device. To fix this we need to create a clean project and add our new or existing native code to that project.

Here are the steps to setup your development environment and create a new, clean project for your native code. Please keep in mind that this guide is still work-in-progress. I may add more detailed instructions and examples later on, when people ask for it.

Update 2011/10/15: Some improvements in the guide, based on comments of rudelm and GoodDayToDie.

Download the attached Microsoft.Phone.InteropServices.zip. After you downloaded the zip-file, open the file-properties and make sure the file is "unblocked" (Windows will block downloaded files). Some unzippers, including the built-in unzipper from Windows will mark the unzipped files as "blocked", which would give problems later on if you don't unblock first.

If your developmachine is 32-bit you go to "C:\Program Files\Reference Assemblies\Microsoft\Framework\Silverlight\v4.0\Pr ofile\WindowsPhone71" or if you have a 64-bit machine you go to "C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\Silverlight\v4.0\Pr ofile\WindowsPhone71". Extract the DLL from the zip-file in this folder.

Open the Visual Studio Commandprompt and change directory to the folder where you just extracted the DLL. Then enter this command:

Code:

SN -Vr Microsoft.Phone.InteropServices.dll

In the same folder there is a subfolder called "RedistList". Open that folder and open the file "FrameworkList.xml". Add this line to that file:

Choose Visual C++ / Smart Device / ATL Smart Device Project and fill in a name and location for your native library. Do NOT choose MFC, or your library won't work on WP7! The name will be the name for the DLL. Later on you will create a COM-class. Choose a different name for your library and for your COM-class!

In the new dialog enter the "Short name" for your COM-class. All other names are filled in automatically. Keep those names default to avoid naming-conflicts. Also make sure the name of your COM-class is different from the name of the library. All other options can are default, so you can click "Finish" now.

The basic layout for your native project is now ready. Note that you have these files: for your library you have a header-file (.h), a code-file (.cpp) and a COM-definition-file (.idl) and for your COM-class you have a header-file (.h) and a code-file (.cpp). I will refer to these files in the following steps, so make sure you can identify these files.

The COM-class you have now is based on IDispatch. IDispatch is the COM-interface that supports reflection-like functionality. The COMBridge in WP7 does not support this interface. Instead we should use IUnknown, which is the base-interface for all COM-objects and supports reference-counting.

In the header file of your COM-class you can see the public inheritance of IDispatchImpl. This is no problem and you can leave it as it is. But you can also see this COM-mapping:

Code:

COM_INTERFACE_ENTRY(IDispatch)

You need to remove that line.

In the IDL file of your library you need to change the inheritance of the COM-class from IDispatch to IUnknown.

Your native code layout is now ready to add your methods. A method in COM-class should always have HRESULT as return-type. This value should be 0 or positive in case of success (normally use constant S_OK for success). If you have an errorcode which should throw a COMException do a logical OR with 0x80070000 and return that value. If you want to return a variable, you'll to declare that as parameter of your method and decorate it as returnvalue in the IDL-file. The parameter-types are bound by the definition of COM. You can read about the supported COM-datatypes here and here. Study those parameter-types closely, because any mismatch in your managed and unmanaged declarations will make your app crash definitely. You need to add all your methods in 3 different places: in the COM-class code, in the COM-class interface and in the IDL-file. Later on you need to add an exactly matching interface to your managed code. All the declarations have their own specific format and decoration. I will give an example of two different functions for these 3 files. Note that in these examples, the COM-class was named "Native", so the class implementation is called "CNative" and the interface is called "INative". You have to change that if your class has a different name.

Locate your interface in the IDL-file of the library. This may look a bit weird, because there are a lot of attributes that decorate the empty interface. Add these declarations to your interface (note the decoration of the parameters, read more here):

Now we need to locate two GUID's and copy them in a text-file, because we need these GUID's later on. These GUID's are in the IDL-file. We will call the first GUID "interface-GUID". It is the "uuid" in the tag RIGHT ABOVE the interface-declaration. We will call the second GUID "coclass-GUID". It is the "uuid" in the tag RIGHT ABOVE the coclass-declaration. There also a "uuid" in the tag above the library-declaration, but we don't need that one.

Open Visual Studio 2010 and create a new project: Visual C# / Silverlight for Windows Phone and choose a project-type, name and location.

Now go back to your native project in Visual Studio 2008. The compiled result DLL of this project will be used in your Windows Phone app. To make sure you always use the latest version of the native DLL in your Windows Phone app, you can add a Post Build Event to this project. This example assumes you will have a folder with a subfolder for the native solution and a subfolder for the Windows Phone solution. Go to Project Properties / Configuration Properties / Build Events / Post-build Events and add this (change the paths according to the soluton-foilder you will create for your Windows Phone app):

Code:

copy "$(TargetPath)" "$(SolutionDir)..\MyApp

If you checked the option "Create folder for solution" when you created the Windows Phone project, you may want to add another subfolder "\MyApp" to the path.

Now build your native project! The compiled DLL should now also be copied to the folder of your Windows Phone app.

Create a new file called "WPInteropManifest.xml" in the folder of your managed Windows Phone app. Copy this content in the file:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<Interop>
</Interop>

Switch back to Visual Studio 2010. In the solution explorer click on "Show all files". Your native DLL and the "WPInteropManifest.xml" should be shown now.

Select the "WPInteropManifest.xml" file and in the file-properties set "Build action" to "Content" and set "Copy" to "Always". You will always need this file in your project, regardless you will be calling drivers or not. If you don't have this file in your project, you won't be able to use your native DLL.

Select your native DLL and in the file-properties set "Build action" to "Content" and set "Copy" to "Always".

In the solution explorer, right-click on the project and choose "Add Reference". Then select "Microsoft.Phone.InteropServices".

Open the "WMAppManifest.xml" file and add this line below the other capabilities:

Code:

<Capability Name="ID_CAP_INTEROPSERVICES" />

Later on, you can try if your app will work without this capability. If you only use native code without calling drivers (directly or indirectly), you don't need the capability and your app will also work on devices that are not Interop Unlocked then. This specific example does not call any drivers, so in this example the ID_CAP_INTEROPSERVICES can be omitted and then it would run on non-Interop-Unlocked devices.

Now add a code-file to your project and copy this code into the file. You need the the coclass-GUID and interface-GUID you copied into a text-file earlier and you also need to replace the name of the class and interface to the names you used. Also note that the declaration must be an exact match (order and parameters) with the declaration in the IDL-file, although the IDL-file is differently formatted.

Now you need to call the native code. You can add this code to the constructor of your Page or to the eventhandler of a button, or anywhere you like. Be sure to replace the DLL-name, interface-name and class-name and use your coclass-GUID. The exception is a well-known error-code and the exception will be casted to a UnauthorizedAccessException, instead of a COMException.

You can now run your project! Be sure that you deploy it to your device. The emulator won't work, because you project uses native ARM code. The emulator runs on x86, so your native DLL won't load in the emulator.

When you go more advanced, you may need the Marshal-class. For example to copy a native memory-block to a managed byte-array. Be aware that there are actually two "Marshal" classes. There is "Microsoft.Phone.InteropServices.Marshal" and "System.Runtime.InteropServices.Marshal". They both look the same. But be sure you are using "Microsoft.Phone.InteropServices.Marshal", because it will allow you to do a lot more! Most methods in "System.Runtime.InteropServices.Marshal" will throw a MethodAccessException, because they are tagged [SecurityCritical], while the same methods in the other Marshal class will work.

I hope this will help you port your homebrew apps to Mango or create some fresh new homebrew! If you created an app with native code, drop me a line here. Show me your Screen Recorders, Accent Changers and more!

Suddenly, awesomesauce! Wow, big thanks Heathcliff74! Eve since you said you'd figured out homebrew native DLLs on Mango, I was really excited to see what people could do. I never guessed the real reason homebrew DLLs didn't work on Mango, although in retrospect this makes sense. You're awesome for investigating this for us.

Thoughts that immediately come to mind:
Update the existing screen capture apps.
Update the existing WebServer app.
(As part of the above) update the sockets DLL so we have server sockets again.
Explore how much filesystem access we have. Can files be copied from one app's isostore to another app's isostore?
Explore accessing drivers. The HTC update breaks filesystem access for HTC homebrew, but maybe there's another driver entry point we can use.
Investigate direct access to the SMS store (message backup?)

looking fwd to the native apps , a universal screenshot apps would be awesome..

Update :
scratch that, just ready that the app will be bound to the rules of the sandbox of your app.I guess that means no universal screenshot app yet

Hi!

Screenshots apps are definitely possible! The API for this can be called from within the sandbox and using OEM drivers it is possible to switch off dehydration. I already discussed this with fiinix and gave him this info. And I believe he almost has a Mango version ready.

Wooohooo nice HowTo! I will definitively try it and will report later. However, that will require that I go back to NoDo and back to Mango first. I'm not looking forward to that procedure... anyways awesome work Heathcliff, thank you!

@GoodDayToDie: you mentioned that the HTC libraries are fixed regarding file access. Julien Schapman from TouchXplorer mentioned something like that a while ago on twitter. Do you have any additional information on that topic? Is it just the DLL files from the HTC apps or is it something with the Mango HTC Update? I'll hope this is reversible, if I go back to NoDo and want to try Heathcliffs instructions :/

@rudelm, I only have experimental knowledge; I haven't dug into the actual update. However, the way that things like ComFileRW.dll work is by calling into some high-permission module in the HTC firmware (probably a driver using an IOCTL, though it could possibly be an RPC call to a privileged process) which then executes the requested action with high permissions. That's why the HTC DLLs don't do anything on other phones; they can't talk to the component that actually does the work.

My guess is that the HTC update simply turned off whatever it was that the COM DLLs are calling into. It could be more complex than that - for example, they could be trying to validate the caller, and prevent it from being used by homebrew - but whatever they did, neither DLL works anymore once you have the HTC update *even though the DLLs themselves did not change.*

Is it reversible? Well, "fixing" whatever component they were calling into is one option. Using Heathcliff74's Root Tools to gain full permissions on a "normal" homebrew app is another. There might be more, but it would need more study.

@rudelm, I only have experimental knowledge; I haven't dug into the actual update. However, the way that things like ComFileRW.dll work is by calling into some high-permission module in the HTC firmware (probably a driver using an IOCTL, though it could possibly be an RPC call to a privileged process) which then executes the requested action with high permissions. That's why the HTC DLLs don't do anything on other phones; they can't talk to the component that actually does the work.

My guess is that the HTC update simply turned off whatever it was that the COM DLLs are calling into. It could be more complex than that - for example, they could be trying to validate the caller, and prevent it from being used by homebrew - but whatever they did, neither DLL works anymore once you have the HTC update *even though the DLLs themselves did not change.*

Is it reversible? Well, "fixing" whatever component they were calling into is one option. Using Heathcliff74's Root Tools to gain full permissions on a "normal" homebrew app is another. There might be more, but it would need more study.

uhoh... sounds pretty bad for HTC users. If it was a firmware update, we will have a bigger problem. I will try to revert back to Nodo and will try Heathcliffs instructions for Native Code first. InteropUnlock is still something I need to try for Mango

XDA Developers was founded by developers, for developers. It is now a valuable resource for people who want to make the most of their mobile devices, from customizing the look and feel to adding new functionality.Are you a developer?