Thursday, December 22, 2011

An Metro Style application needs to be packaged, signed, and deployed before it can be launched from the the Windows desktop, as I discussed in my post on Creating an Appx Package. This means an IDE needs to go through a lot more work in order to launch an application for debugging than just calling CreateProcess with the DEBUG_PROCESS flag set.

Creating a package

For now, I'm just using createappx and the PowerShell cmdlets Add-AppxPackage and Remove-AppxPackage to manage my packages from batch scripts. IDE integration should use the App Packaging and Deployment APIs to achieve these steps.

IPackageDebugSettings

Windows Runtime internals: understanding "Hello World" is a talk I've mentioned before as a must watch talk. A large part of what he does is debugging low level stuff with the provided command line debugger, cdb. I found this indispensable for getting started. In this talk he uses a tool that enables debugging on a package using IPackageDebugSettings. He shared this tool unofficially on codeplex, and then Microsoft packaged an official sample on their own site. Another package Matt suggested you get is the Windows Developer Preview Symbols (for x86 and x64), which are available for download through MSDN. These symbols are useful for debugging the internals of the Windows libraries from cdb, windbg, or Visual Studio. It's handy being able to put breakpoints in things like, RoGetActivationFactory to see a little bit of what's going on behind the scenes in WinRT.

Something that is not obvious to me about the IPackageDebugSettings.EnableDebugging API, I'm not sure if you can specify the parameters. When I specify a debugger, it seems to add the parameters "-p (PID) -t (TID)", and I'd like to know if I can tell it to provide those in a different way. With RAD Studio, the command line argument to attach to a debugger is "bds.exe -attach:(PID);(TID)" and in order to parse the "-p (PID) -t (TID)" I need a helper script/program.

Automating the startup

I'm still missing a part of automating my application startup for debugging, and that's actually launching the application. I found an article by Raffaele Rialdi on contracts and activation where he describes how Explorer launches a process:

"Explorer uses the RPCSS (RPC system service) to activate the application. As RPCSS is exposed to the network and has a large attack surface, RPCSS runs with low privileges (network_service) and with a integrity level system that prevents other processes with a lower integrity level can affect RPCSS in reading or writing.

As we have been told during the build conference, RPCSS maintains the list of activated objects, and can create a pool of objects (anyone remember COM+?) via IObjectControl.

To start the process (real or surrogate), RPCSS service uses "DCOM Launcher" that has higher privileges (localsystem). The "DCOM Launcher" service reads from the registry the informations related to the class and the host process and finally create the application process."

This all seems like relevant and useful information, I'm just unsure how to do that from my own application. Or possibly there's some way I can tell Explorer to do all that?

Debugger woes

Before I mentioned attaching my application to RAD Studio for debugging, which is unfortunately something I've hit a bit of a road block with. I actually have the same problem if I debug with the x86 version of cdb.exe or windbg.exe. The debugger will attach fine, but when I attempt to run, I occasionally see an Access Violation that the application can't recover from. The Access violation happens in the loader before my exe has even been loaded, so I suspect this has something to do with new linker features for Win8 in cl.exe and csc.exe; there's probably some information I'm supposed to add to my exe file that the Delphi compier can't do yet.

Oddly enough, if I attach with the x64 version of cdb.exe (or windbg.exe), I still hit the access violation, but the debugger more gracefully handles it and the application continues to load and run. I can even detach from that debugger, re-attach with RAD Studio, and debug my code in a more familiar environment.

Unfortunately, for some reason the loader loads the executable image with the name "image(RANDOMBASEADDRESS)" and it doesn't actually load symbols. I can force RAD Studio to load the symbols by opening the Modules tab (ctrl+alt+m), right clicking my module selecting "Reload Symbol Table..." and selecting my the executable or its TDS file on disk. It's very clumsy, but for now, it works.

Wednesday, December 21, 2011

Windows 8 Metro Style apps can either use XAML as I discussed before, or they can use DirectX. This is great for Delphi because our in our newest framework FireMonkey the canvas is powered by DirectX on Windows platforms. So once we can create a DirectX, we should be able to turn that into a FireMonkey canvas, and the full power of FireMonkey should be available to the Metro Style application.

Let's look at what Visual Studio generates when you specify a new DirectX application, and try to recreate that with Delphi. Earlier I looked into creating an application using Windows.UI.Xaml. Application, but in order to use DirectX, you must activate the lower level Windows.ApplicationModel.Core.CoreApplication class, and construct your graphics device and surface from the ground up.

As an aside, internally, the Xaml Application has a CoreApplication as well. At //build/, Matt Merry gave a talk on Windows Runtime internals: understanding "Hello World" where he shows the internals of a simple hello world Xaml application, debugging down to the CoreApplication level. This is a must watch talk if you want to truly understand the inner workings of a WinRT application.

Back to Delphi. Like my first post about Xaml, this is all pretty rough and I don't think the code is ready to share as a complete project. But I'll discuss the finer points of most of it.

There's a bit of a difference here between the Xaml application and this one. Here, we're accessing ICoreApplicationInitialization from Windows.ApplicationModel.Core.CoreApplication. ICoreApplicationInitialization represents a set of static methods that is available for this class -- there is no instance of the CoreApplication object. Instead, we just need to provide the CoreApplication class with a IViewProviderFactory which knows how to create a runnable view. My implementation of IViewProviderFactory is simple:

Once the application successfully gets a ViewProvider, It calls it's Initialize providing a Window and an ApplicationView. It then calls Load, specifying an entryPoint. This entry point is associated with Windows Application Contracts, and for now we're just looking at the launch contract. The entry point for the launch contract is going to be the entry point you specified in your appxmanifest, in the Application node. Here I've specified 'DirectXApplication.App'. After it's been Loaded, Run gets called, and we construct and run a View.

Here we see that the view plays a similar role as a message loop in a Win32 application. As long as we're running, we tell the application to process events that have occurred, update the render target, and finally present it to the screen. Here I'm using System.Diagnostics.TStopwatch to notify the render target how long it has been since the last update. Microsoft's DirectX Application template does something similar with QueryPerformanceCounter, which is actually the same thing TStopwatch does behind the scenes, although TStopwatch has the benefits of falling back on lower resolution timers if no high resolution is available, and it is available cross platform. It might be useful in your apps as well.

The details of my TD3DRender are pretty standard Direct3D calls, create a device, render target, back buffer stencil, swap chain, back buffer texture, and the only painting I'm doing is clearing to blue. The code is essentially a direct translation of everything in the Visual Studio's template D3DRenderer.cpp, and I don't think it adds anything by including it here. The only new thing is there's a new method on IDXGIFactory2 for D3D11, CreateSwapChainForImmersiveWindow. In Metro Style apps, you don't have access to Window handles, so we can't create the render target in with the usual arguments; instead we have to specify the IWindow.

With that, I have a DirectX application written in Delphi. One thing I noticed is I cannot run it as a normal executable the way I could with the XAML application I made. The call to Windows.ApplicationModel.Infrastructure.CoreApplication.Run(factory) fails with HRESULT $80004005, "Unspecified error". I'm not sure what happens differently when running from the Desktop, but I assume there is some initialization that doesn't happen. So in order to actually run this, I had to package and install it. I suspect it might be related to new compiler/linker features being needed for Windows 8. csc.exe and cl.exe both have new options specific for Windows8. "/t:appcontainerexe" to build an Appcontainer executable makes me think there might be something different they're linking in as a hint to Explorer; that's clearly not something the Delphi compiler is doing but something we need to investigate.

Monday, December 19, 2011

One of the main design goals of the Windows Runtime is to provide a broad range of APIs that can be consumed across a variety of technologies and languages. In fact, everything designed for WinRT can be made accessible to other languages through the use of language projections. Every class Microsoft provides and every class you write in C++ and C# in Visual Studio generates .winmd files that describe to other bindings how to use that. You simply need a tool that turns those meta data files into code in your language of choice. I suggest you take a look at Microsofts documentation on how to Consume and Emit .winmd Files to get some direction on where to begin with some other language, such as Delphi.

Okay, that was a little tongue in cheek; At the time of publishing this article, that MSDN article was essentially blank, a stub for future documentation. But some other MSDN articles did give me some more clues. The overview of the Windows Runtime design describes the .winmd files: "The Windows Runtime is exposed using API metadata (.winmd files). This is the same format used by the .NET framework (Ecma-335). The underlying binary contract makes it easy for you to access the Windows Runtime APIs directly in the development language of your choice." And if that wasn't enough the new APIs to query for meta data of APIs (RoGetMetaDataFile) return a IMetaDataImport2 and a mdTypeDef. These are exactly the same things you need to inspect metadata provided by .NET assemblies, which you can already do in Delphi using the APIs in Winapi.Cor and Winapi.CorHdr (which come from cor.h and corhdr.h in the platform sdk).

Another great tip I found somewhere, because the metadata files are just Ecma-335 metadata, they look just like .NET assemblies to tools such as .NET Reflector, and Microsofts own IL Disassembler. So out of the box in Windows 8, you can use ildasm.exe to peek at the contents of .winmd files. In the image to the right here, we can see the metadata in Windows.UI.Xaml.winmd. Every class is described in detail: what interfaces it implements, what methods and properties and events are available. Every enumeration and structure is described in detail as well. All that needs to be done now is develop the tool that turns this into Delphi interfaces. While I'm not going to go into the specifics of this tool, I'd like to point out some follies I fell into when translating these files.

Namespaces

The first thing I noticed is that Windows is doing a lot to break the API down into Namespaces, and for the most part each metadata file describes one namespace. So my first reaction was to put the contents of each metadata file into one Delphi unit. This doesn't quite work though because of a few things: One metadata file may actually contain multiple namespaces, one namespace might be included in several metadata files, and there are circular dependancies between types in the various metadata files. Since in delphi one unit = one namespace, I've lumped everything into one big unit that exposes all of the Apis defined in the winmd files provided by Microsoft to work around this issue for now. If you've looked at my code on git, you'll also notice that the dotted namespaced types have been flattened, using "_" instead of ".". Nesting the classes even within this one large file becomes difficult because of the interdependencies among classes and the need for things to be defined before they can be used. A smarter tool should be able to create an tree of all the dependancies and actually construct a more OO mapping similar to the C++, C# and Javascript ones.

Classes

Classes from the point of view of Delphi are exposed in much the same way as COM objects, as interfaces. You tend to create them with RoActivateInsance, which is similar to CoCreateInstance, or RoGetActivationFactory. For classes that implement "static" methods, these methods are acquired by requesting an interface. For example, Windows.UI.Xaml.Application implements Windows.UI.Xaml.IApplicationFactory (for constructing an Windows.UI.Xaml.IApplication) and a Windows.UI.Xaml.IApplicationStatics. A Delphi wrapper will need to implement these likely by storing the reference to the interface in a class private field and implementing each method. I imagine something clever might be possible using RTTI to define class static interfaces and do something similar to dynamic aggregation (the implements keyword), injecting the interface into the class VMT.

Enums

WinRT Enums will almost certainly need to be scoped. There are enums with the same name in different namespaces, and it's very possible for the enum values to be different, especially as more third parties add their own. Microsoft ordered the values of Windows.Graphics.FontStyle (Normal, Italic, Oblique) differently than Windows.UI.Xaml.FontStyle (Normal, Oblique, Italic) for example. It shouldn't be an issue if they're scoped; the compiler will type check and it won't be a problem.

Events

Something that caught me off guard: the interfaces for Event handlers don't inherit from IInspectable, dropping the pattern that "Everything is IInspectable". This seems somewhat weird; I thought that I should be able to implement say, Windows.UI.ActivatedEventHandler on any class I want, and since it's an interface, WinRT will do QueryInterface to get the correct interface before calling Invoke. This isn't the case as all; QueryInterface is never called for EventHanders, it just assumes you're giving it the right one. Presumably this makes it faster to call the Invoke behind the scenes but it seems to me a single QI at the beginning wouldn't be too much of a hit on performance. So EventHandlers all inherit from IUnknown. In my previous post I have an example of a wrapper class for an event that takes an arbitrary procedure reference; this is likely going to be a good way to create event delegate objects.

Generic Interfaces

Generics are used pretty heavily throughout WinRT. In the C++ language bindings, they end up translating down to a template class, and the pre-compiled headers actually have to parametrize every instance of a generic class. This is pretty ugly and it'd be much nicer to just declare the interface and have the compiler do the right thing. Delphi has a similar problem for similar reasons. Generic Interfaces are supported by the compiler, but the problem arises in that you cannot specify a GUID for parametrized instances of a given generic interface. So for now, I've had to create a unique interface for every parameterized interface in WinRT. But how do you go about assigning a GUID to the interfaces? The answer lies in RoGetParameterizedTypeInstanceIID.

Lets take a look at the IL of something that uses a generic. How about Windows.Storage.FilePropoerties.IStorageItem.ExtraProperties.SavePropertiesAsync:

Actually let me touch on those attributes since that's sort of interesting. There is an OverloadAttribute, which says this method overloads another method named SavePropertiesAsync. This is sort of interesting because in C++, the methods can't be overloaded with the same name: This one is SavePropertiesAsync, the other is SavePropertiesAsyncOverloadDefault, as described by it's own OverloadAttribute. This is useful information if your language doesn't support overloading methods with different arguments. Delphi doesn't have that problem, since there are different arguments, we can name them both the same.

Okay, so the method has one parameter propertiesToSave, of type Windows.Foundation.Collections.IIterable`1<Windows.Foundation.Collections.IKeyValuePair`2<string,object>>. If you're not familiar with IL, the "[Windows.Foundation]" denotes that the type is a typeref, and the complete typedef can be found in the assembly (or in this case, metadata file) Windows.Foundation. The `1 and `2 after the generic type names is an indication of how many generic arguments there are. To me this seems like an odd convention because it should be given how many arguments there are going to be, and it's strange that it's called out in every type. But it's a clue to some of the new APIs, such as RoParseTypeName which will validate if a type name is valid, given the number of generic parameters, and for the IRoSimpleMetaDataBuilder sink for actually getting the GUID. In order to use RoGetParameterizedTypeInstanceIID, we need a class that implements IRoMetaDataLocator, which uses the Locate method as a call back to query for more information about "unknown" types. It actually ends up querying about every Interface type, but knows how to handle basic types such as "int", "string", and "object". Remember, in the context of WinRT, a string is a HSTRING, and a Object ends up being some class that implements IInvokable. But for the purposes of this API, they're "string" and "object".

So we need a class that implements IRoMetaDataLocator. There's a pretty big gotcha here that caught me off guard. The documentation for IRoMetaDataLocator states "The IRoMetaDataLocator interfaces inherits from the IUnknown interface." That statement isn't surprising, but what is surprising is it's absolutely false. Look at the definition from roparameterizediid.h:

Looking at it now, potentially I should be using a record to more closely match the C++. IRoSimpleMetaDataBuilder actually has the same issue: there is no IUnknown members for that interface.

As I said before, the locate method gets called with each part of the type that is being fetched, so the first time it's invoked, we get "Windows.Foundation.Collections.IIterable`1", the second time we get "Windows.Foundation.Collections.IKeyValuePair`2". We don't get called for "string" or "object" because it handles them internally. My Locator needs to lookup each string and provide the metaDataDestination with it's guid, and specify if it's generic or not. I'll give a sneak peek at that code, although it's using some helper classes that aren't publicly available to make using the Cor interfaces easier to use.

Note there are other types that could be passed in to a parameterized type, Enums, Delegates, or Structs. I'm ignoring these for now because I didn't actually encounter any API that uses those types.

The somewhat problematic thing about this is this might be difficult to include in a compiler, because the APIs are only available on Win8, so there's no way to generate the GUIDs from a older version of Windows. It'd be nice if Microsoft provided information on how those GUIDs are constructed so other tools that run on other Windows platforms can cross-build for Win8.

I hope this helps to de-mystify some of the information provided in the Windows WinMD metadata files.

Friday, December 16, 2011

I promised before I'd try to get this code up somewhere. I've uploaded this entire example to github here. Please read the notes in the Readme about deployment, it requires signing the package with a certificate you've added to your trusted root stores.

The app I discussed in my previous post is fairly boring: all it does is present some text to the screen. I wanted it to do a little bit more to experiment with how events are attached to controls in WinRT. How about I make the text change color when the mouse hovers over it? Actually I shouldn't say mouse, I mean Pointer. In Xaml apps, mouse, keyboard, and touch events are all unified into a Pointer events so they all operate in similar ways. Anyway, it is a simple addition that makes the app more interesting.

WinRTs notion of events is a little different from VCL, and I think it'll be a welcome change. Events are handled with multicast delegates, similar to in C#. With any given event for any control, you could add multiple delegates to listen for that event. Each event has an add_* and a remove_* method to install or uninstall an event listener. For my Text, I'll be adding PointerEntered and PointerExited events. Both of these require a Windows.UI.Xaml.Input.PointerEventHandler, so I'll create a little delphi wrapper for that class.

This creates an instance of a handler, and stores of a reference to a procedure that should handle the event. That means the code for my event handler can still live in TDerivedApp, even though it's handed off to WinRT by this helper class. I've added an OnPointerEntered and OnPointerExited methods to my application class. In my OnLaunched event is where I actually hook the event handler up to the code:

I've omitted the code for OnPointerExited here; it's nearly identical to OnPointerEntered except it uses a white brush. Again, some of the code here, like creating a brush, is quite verbose and ugly; eventually class wrappers will make this look much nicer, similar to the C# and C++ projections that would alow this to look something more like:

Thursday, December 15, 2011

In order to turn an application that runs in windowed mode into a full screen Metro application, I had to package it and deploy it. There are already a few places with lots of information about what's required for this, so I'll just go over the important steps I gathered from the following:

Once it's running on the Desktop fullscreen, you might want to hook up a debugger. For that you need to tell it you intend on debugging. IPackageDebugSettings allows you to enable, disable, suspend, or resume debugging a package. Microsoft has provided a command line utility wrapper around this class that allows you to do this easily.
If anyone can provide me information on how to get a process (for example, a debugger after you've used IPackageDebugSettings to enable debugging) to launch a Metro app the same way Explorer does when you click it on the desktop, I'd appreciate it.

Wednesday, December 14, 2011

There is relatively little information out there on developing Native Metro applications, especially for languages other than those that Microsoft has provided projections for (C++, C#, and JavaScript). Other languages, including Delphi, need to create their own projections before they can use them effectively. I've been investigating getting Delphi to produce native applications for WinRT. This is a rough low level look at the WinRT native APIs, and how to use them from a language that isn't C#, C++, or JavaScript. Eventually this investigation will lead to development of Delphi wrappers around the Windows 8 WinRT classes that will be much easier to use. Much of the discussion in this post is very similar to Ian Griffiths discussion on Real Native WinRT Development, but where he's using C++, I'm using Delphi. If you're interested in more technical detail about what's going on behind the scenes, I suggest you give that article a read.
I started with an empty console application, and removed the {$APPTYPE CONSOLE} directive. Essentially this just allows me to have a Windows application that doesn't use VCL. Eventually this application will need to be wrapped up as a AppX package, I'll discuss that more in depth later. It seems that WinRT XAML applications can be run outside of Metro, so for now I'm take advantage of that.
I'm going to jump around in the code a bit, sharing important as it comes up. Lets start with the uses clause:

program SampleWinrt;
uses
System.Win.ComObj,
Winapi.Winrt in 'Winapi.Winrt.pas',
AllWinRt in 'Generated\AllWinRt.pas',
WinRtHelpers in 'WinRtHelpers.pas';

There are three new units here:

Winapi.Winrt contains translations of the new global WinRT APIs. These are essentially the functions and datatypes defined in the Windows Runtime C++ reference: things such as RoInitialize and WindowsCreateString, etc.

AllWinRt contains all the enumerations, records, and interfaces defined in the Windows Metadata files. This file was generated by a tool that turns the metadata into delphi style interfaces. I'll go into the details of more of this process at a later time.

WinRtHelpers contains some helper classes that reduce the amount of code one needs to write in order to use the native APIs. This is the starting of a Delphi wrapper around the API to make the code more "Delphi". Currently, it contains a TInspectableObject-- which similar to TInterfacedObject but also implements the IInspectable interface -- and TWindowsString -- a helper class to convert strings to/from HSTRINGs.

Next up, we need to initialize our application. So lets look at the main program body:

This code is just to initialize the Windows Runtime, and make sure it gets uninitialized on termination. RoInitialize is similar to COM's CoInitialize -- all threads that interact with WinRT objects must call RoInitialize prior to using it. The application can be initialized as single threaded or multithreaded. Ian Griffiths had difficulty getting the RO_INIT_SINGLETHREADED and likewise my app seems to have issues starting up as single threaded. I suspect this problem may just be a bug in the Metro API.

There are a couple of Application Runtime classes in the WinRT framework: Windows.ApplicationModel.Core.CoreApplication, and Windows.UI.Xaml.Application. The Xaml Application is easier to set up, so we'll start with that one. It is interesting to be able to create Xaml applications in Delphi, because immediately we can take advantage of all the Xaml controls provided by Microsoft as well as consume third party controls (regardless of implementation language) and even provide libraries of additional Xaml components that could be consumed by Delphi or other language projections.

Here we create a TDerivedApp (which I'll expand on below), request a Xaml ApplicationFactory, and construct a IApplication specifying the TDerivedApp is the outer implementation of IApplicationOverrides. The ApplicationFactory provides an Inner implementation, which we store off so that we can make inherited calls. Finally we Run the app. This is sort of a bizarre inheritance model provided by WinRT that I'll gloss over here. Ian Griffiths has an in depth article about how to create inherited WinRT classes, give that a read for some of the more technical details of how it's implemented in C++.
Everything here is being stored in Interfaces, so it's reference counted. Even our classes, such as TDerivedApp are reference counted. TDerivedApp derives from TInspectableObject, which derives from TInterfacedObject from System. The Delphi compiler automatically generates calls to _Release, making this code look a lot cleaner than the raw C++ version. Note however that App is being initialized to nil manually before exiting. If Inner is released first, WinRT crashes. I haven't investigated closely, but my suspicion is that Inner may actually be a weak reference to the same object as App, and releasing both is unnecessary/incorrect.

So what is our TDerived app? It's an object that implements Windows.UI.Xaml.IApplicationOverrides so we can actually define the behavior of our application.

Note, like in C++, we have to implement all the entries in the interface, even if we don't want to handle those events. Implementation of those methods will just be a pass through to the Inner IApplicationOverrides provided by the Activation Factory. For example:

Eventually there should be class wrappers that handle all the inheritance for you if you don't want to override methods. When the Application Runs, we get an OnLaunched event. That's the one event we wanted to hook up, so let's do that:

Here we just load some constant XML data into the current frame using an XamlReader. It would be easy to design your Xaml in a separate file using Rad Studio or your favorite XML editor and storing on disk it along side your Application, but for simplicity's sake I've just included it inline.

And that's what it takes to present some text in Metro Xaml controls. Obviously I've omitted some code here; I'll see if I can minimize the AllWinRT and Winapi.Winrt units to only the APIs used in this example to provide a full code sample.