Motivation

I recently worked again a bit on ezEngine, a beautiful Open Source game engine developed continuously by a few friends of mine in their spare time. ezEngine uses custom implemented formatting function that follows the style of (s)printf. Like printf it is C-varargs based and therefore not type safe. Example:

printf("%i", "not an int");

Depending on the printf implementation this will lead to ugly runtime errors. However, all modern compilers have hardcoded compile time checks for that and will warn you. If you have your own implementation but with a similar interface, the story is different though.

Using modern C++ and variadic templates, you could implement all sorts of nice interfaces that give you clear error messages at runtime – still not trivial, but straight forward compared to what follows here ;). But the goal of this exercise today is different: We want to write code that has no influence on the runtime and works with any existing printf-like interface.

There have been more complex things done at compile-time, so how hard can it be to do a bit of parsing with now that we have constexpr…

Compiler targets are vs2015 and the newest versions of GCC and Clang with -std=c++11.

Before we start – on constexpr

Conditions

Using constexpr one could think that we can just loop over the letters of the incoming literal and that’s it. Well not so fast! Loops and if conditions in constexpr are C++14 features. But we are allowed to use the ? operator and recursion (almost) as much as we want!

Throw & Error Handling

There is a special way of exiting a constexpr at compile-time: throw. Yes. Just throw an exception at compile time! But again, this is only allowed in one of the branches of the ternary ? operator. Single standing throw expressions are a C++14 feature. Throw is very helpful for our task, since it allows us to both exit a function and document the place of failure.
Since we are evaluating at compile time, it sounds at first that we can do error handling with static_assert. This doesn’t work though, since static_assert is executed at the template instantiation phase which means that it can’t look at values that are passed to the constexpr function – remember, the function may be called at runtime as well!

Passing arguments

Passing any arguments other than the format string literal and the current parsing position is very tempting since it would allow us to infer types automatically. But depending on the compiler you will get complaints about the non-constness of your constexpr call earlier or later. Instead we need to get the types from the expressions (using decltype) and pass them in the template type list. If the incoming string is not a compile time array, we obviously cannot call a constexpr function at compile-time that tries to inspect it.

Function Evaluation

MSVC2015 is super lazy when it comes to compiling constexpr: Sometimes only branches that are hit will be compiled. I didn’t realize that for quite a while which was a major pain since I based everything on template functions instead of structs which can be easily specialized for all, even unreachable, cases.

This piece is the general entry point and contains already most of the important principles. It is a specialized struct which allows us to write fallbacks for every special case the compiler may stumble onto.
Note that the Entry function expects you to pass at least a void argument. This is a little trick to work around the fact that the macro we’re going to use does not handle zero parameters very well (more on that later).
Also, Entry provides us with a nice way to have a fallback overload for cases where we can’t do compile time parsing (iff the string is not known at compile time).

All the action is in the Recurse function. We use an enum class error code for both throwing (cancel) and return. This makes the control flow very easy and allows us to put the whole thing in static_assert at the very end. The Recurse function will either.. well.. recurse, or branch out to a parse symbol function which follows a similar pattern and will eventually lead back to the Recurse function.

Note that we already use the typename Param0, typename ...Param pattern even if we are not yet reducing the number types. This is just a useful and clean way to make clear that we have at least a parameter type left – writing just typename ...Param would also allow empty parameter packs! The no parameter specialization is basically the same, but without the Entry functions and the ParseSymbol call:

Note that the “no-paramter specialization” for this is unreachable and Visual Studio doesn’t even need it! Every other compiler that clings closer to the standard though will have trouble if you don’t implement template struct ParseSymbol.
As you can see, we pass the FormatType type on to another function called CheckArgument::CheckAndContinue. Also, we need to do some skipping for the various more complex format cases – there is a lot of untapped error checking potential left here!

How to tell if a given type is compatible with a FormatType? With template specialization:

This is also finally the function that “eats” parameters from the variadic template.
To reduce the number of needed ParamCheck specialications, we use these tow little helpers from the standard libraryto get rid of const/volatile and reference in the type:

The rest… not so much. I’ve ended up employing a mixture of macro concatenation and __VA_ARGS__ counting to wrap every single parameter into a decltype instruction. All this is quite complex and since this blog entry is already too long, I refer you to the code and this nice list of macro tricks. As this technique doesn’t allow 0 parameters, I added void() (decltype(void()) is void) as an extra parameter into the macro (which is then removed by the Entry function again).

The caller macro expands things that have been one line to two lines which breaks existing one lined if/else blocks. One can easily work around this by replacing with do { /* all the stuff */ } while(false) but this won’t work if your original function was a member or namespace function which is always the case for ezEngine

Still a nice learning experience

]]>https://wumpfblog.wordpress.com/2016/09/17/c-compile-time-checked-printf-format/feed/2wuempftlbruempftlVirus X Monogame/UWP Porthttps://wumpfblog.wordpress.com/2016/08/13/virus-x-monogameuwp-port/
https://wumpfblog.wordpress.com/2016/08/13/virus-x-monogameuwp-port/#respondSat, 13 Aug 2016 15:03:53 +0000http://wumpfblog.wordpress.com/?p=326Continue reading Virus X Monogame/UWP Port]]>I finally finished porting an old classic of mine from XNA to Monogame and into the Windows 10 store: Virus X, a colorful party game like arcade game for up to 4 players.

You can find the source, more description and download links on github.

Why port?

Couldn’t do any fixes anymore because XNA only works with old Visual Studio versions

User needs to install redistributables

Difficult to put in Windows Store

A few notes on the difficulties during the port:

I tried the whole thing two years ago already, Monogame has grown a lot since then, so they were not a lot of obstacles

Shaders needed some extra care..

XNA is DX9 but Monogame is DX11, so all shaders were previously model 3. ps/vs_4_0_level_9_1 usually did the trick, but there are some minor differences

The single most complicated thing was Texel/Pixel offsets:
All particle movement in VirusX is computed on the GPU using textures for their data and pixel/vertex shaders since DX9 didn’t have Compute Shader (you can’t use them with Monogame despite it using DX11 since Monogame is still extremely close to XNA4 ). However, pixel coordinate systems are different in DX9 and DX10/11! MSDN provides a good overview here.
By the way: As if the Compute Shader limitation wouldn’t be enough, XNA/Monogame has no point drawing. To add new particles to the data textures without perform a blocking read + write (no async read/write!) I needed to draw single-pixel wide lines.

All Asset settings need to be ported manually. Not a big deal with Monogames Asset Library tool though.

And for UWP (Universal Windows Platform == Windows 10 Store):

SharpDX.MediaFoundation.dll and thus Monogame.MediaPlayer does not work! You need to port all your music playing to something else.

I used Windows.UI.Xaml.Controls.MediaElement which is quite easy to use.

Your window is always resizable. This led to quite some headache in Virus X which relied on resizes only in the options menu. Doesn’t work everywhere properly. You can kindly ask the UWP view for a preferred min/max window size (ApplicationView.GetForCurrentView().SetPreferredMinSize(...)), but that didn’t work for me at all.

File access in UWP is very restricted (in a good way from the user’s perspective) and you’ll find yourself suddenly without your favorite .NET file functions. Also, most of it is async.
Luckily I didn’t have a lot of them in there.

XML reader apis are different but quite similar

Localization…
The original used resource files (resx) for strings. These files generated a class that provided all the strings. There is a very similar mechanism in WinRT, but it it differs in the details. The resource file (resw) has the same format but needs to be in a given folder structure. There is no class generated, instead a Windows.ApplicationModel.Resources.ResourceLoader will always provide you with resources for the current app language. Setting the app language proved surprisingly tedious. Without discussing the details here, I’ve ended up with this code:
var culture = new CultureInfo(value == Languages.English ? "en-US" : "de-DE");
Windows.Globalization.ApplicationLanguages.PrimaryLanguageOverride = culture.Name;
CultureInfo.DefaultThreadCurrentCulture = culture;
CultureInfo.DefaultThreadCurrentUICulture = culture;
Windows.ApplicationModel.Resources.Core.ResourceContext.ResetGlobalQualifierValues();
var loader = new Windows.ApplicationModel.Resources.ResourceLoader();

The switch from generated class to something like VirusXStrings.Instance.Get("MyString") was quickly done via a regex-replace. I considered using a dynamic class to keep the “member semantic” of the previously generated code, but after some problems I decided against it.

Submitting to the store itself is pleasantly uncomplicated. Just use the export package function from Visual Studio, create a store page with a Microsoft account and work through all the various options of a submission.

Once submitted, you can even see usage statistics and crash reports. However, the crash reports I got in so far didn’t help me fixing their causes – apparently there is some loading issue on startup on some machines.

Recently I ran into a weird situation where a threading lock was either passed, or not (since locking might or might not be appropriate in a given situation). Since I am always very concerned about other people adding early out return statements and forgetting to do the cleanup (here: unlocking the lock), I wanted to use a scoped version of the lock. But having the scope optional forced me into using an allocation:

This certainly solves the problem – the lock is still scoped and nobody needs to worry about having the lock unlocked. Way better than checking the lock variable at each return point. But it requires an allocation and no longer keeps the ScopedLock on the stack as it is intended to be, how awful!

Template programming for the rescue! Using variadic templates and placement constructors we can solve this problem in no time once and for all!

Scoped thread locks are of course only an example. The template can be used with any kind of scoped object that may or may not be required.

All this almost let forget you about the simple solution: Your scoped object should take a pointer and it should handle null. (yeah, I forgot about that myself when I had this whole idea, thank you for pointing that out to me @ Christopher)

]]>https://wumpfblog.wordpress.com/2016/04/30/microproject-optionscopedobject-template-class/feed/0wuempftlbruempftlAbstracting (DX12/NewAPI) Resource Transitionshttps://wumpfblog.wordpress.com/2015/12/13/abstracting-dx12newapi-resource-transitions/
https://wumpfblog.wordpress.com/2015/12/13/abstracting-dx12newapi-resource-transitions/#respondSun, 13 Dec 2015 12:15:20 +0000http://wumpfblog.wordpress.com/?p=273Continue reading Abstracting (DX12/NewAPI) Resource Transitions]]>The new generation of low-level rendering APIs introduced the concept of “Resource Transitions“: The API user is responsible for keeping track how a memory resource is used and make explicit transitions in the command-list. Most common example: Use a texture as rendertarget and vice versa.
(This is definitely true for DirectX12 and Mantle, but I do not know for sure about the upcoming Vulkan-API… although it would surprise me if they handle it differently)

Need to create command lists on the fly (or pre-cached) – how costly is that?

Record command lists multi-threaded in custom data structure without sending any commands to the API. When all lists are ready, missing resource transition infos can be propagated. Only then the API-side command lists are generated

Has anybody tried that?

Pro

Manual recording needs may be necessary to emulate multi-threading with old APIs anyway

Con

Requires multiple threading barriers

Two step recording introduces a lot of overhead

In DirectX12 resource transitions are a special form of Resource Barriers.
Resource barriers in general were already in place in the old APIs, e.g. the infamous glMemoryBarrier. A common usage example are unordered access views, where the question weather caches need to be flushed or not depends entirely on the shader operating on it (usage example here). None of the discussed approaches is able to abstract such UAV Barriers entirely away, without generating too many of them. More or less the same goes for Aliasing Barriers (different resources on the same memory).

For the ClearSight project I am torn between the “no abstraction” and the “command list fill-in” which sounds reasonable to me (and hey, a guy on the official DX12 edu channel told us about it!). Automating the process or resource transitions sounds very attractive is probably a must-have for every serious render-abstraction. However, I don’t think it is trivial to implement in a robust and fast fashion. And as I mentioned it cannot abstract all resource barriers.

So for now I will just pass the burden to the user. I am curious how much extra work this actually means to the implementer of a high level rendering pipeline. I suspect that it should be fairly simple since most transitions are made deliberately. Assuming the API debug layer catches all missing transitions, it should also make it easier to see where unnecessary overhead was introduced:
An automatic system may fill in all transitions, but does not report where the graphics programmer did not intend to perform transitions.

]]>https://wumpfblog.wordpress.com/2015/12/13/abstracting-dx12newapi-resource-transitions/feed/0wuempftlbruempftlAbout Resource Handleshttps://wumpfblog.wordpress.com/2015/11/28/about-resource-handles/
https://wumpfblog.wordpress.com/2015/11/28/about-resource-handles/#respondSat, 28 Nov 2015 15:29:09 +0000http://wumpfblog.wordpress.com/?p=37Continue reading About Resource Handles]]>For the sake of this article I define a resource as an abstract object that cannot or should not be created, destroyed or manipulated directly like you would access a variable. Usual examples are objects that are associated with “external memory” like files or GPU-textures.

Many game engine do not allow access to resources by (raw/ref-counted/etc.) pointers to class interfaces, but through special resource handles that don’t have any special member functions at all. A resource handle is usually an identifier which is used together with a resource manager to either:

perform operations on or with it

temporarily retrieve an interface to the resource (pointer to a class instance) that represents or even holds the actual data

I think these two two possibilities are important to distinguish. I call them indirect and direct resource access (beware! I made these names just up). Note that you can easily map both access types to each other, so in the end it is more about programming interfaces!
The indirect resource access is fairly common, especially in non-object-oriented languages like C, where it is not possible to give pointers to resource-interfaces. A good example for such indirect access handles are C’s file-handles.

Either way, why should one use resource-handles in object oriented languages at all?

The resource system can change how a handle is resolved. This is useful for:

resource fallbacks

not found

still loading

streaming

uncertain memory location

might even be used for detail levels

…

They act similar to weak pointer

Resource can be destroyed explicitly:
Subsequent queries (direct access) or function calls (indirect access) at the resource manager notify the user that the resource does not exist

It is all about moving power from away from user handled “resource instances” to a resource system which has global knowledge.
If you cannot make use of the mentioned advantages, just go with the normal object oriented approach where you create/destroy/manipulate at any time without special supervision. The disadvantages of a handle system are rather obvious:

Additional indirection

[only direct] Resource system can be circumvented by just asking only once for the resource and keeping it then

Oh so there is a disadvantage for the direct resource system but none for the indirect? Not so fast: Giving a resource object/interface to the user allows much nicer usage and can leverage polymorphism. In contrast, the indirect system needs to have all functionality in the manager which can be very unnatural and inflexible.

Of course all this is very simplified and general. In some situations a hybrid might be useful as well!
More infos about “indirect” handle-based resource managers can be found for example here. An example for a “direct” handle-based resource manager can be found in the ezEngine.

Now to the practical usage in ClearSight, my current C# rendering framework (as a precaution I stopped calling it “Engine” ). So far I have identified two places which are in need for some kind of resource system with different properties.

(low-level-ish) Renderer

ensure that resources that are still in use by the GPU (!) are not deleted immediately

few different resource operations

resource provides very few operations itself

creation can be assumed to be very fast; only states are thus either existing or not existing

higher level asset system

load by lookup (do not reload loaded resources)

fallbacks

many and complex resource operations

possible multi-threaded loading processes, different loading states

Since in C# any object that is still referenced somewhere cannot be deallocated, the advantage of a weak-pointer-like behaviour is irrelevant. Needless to say that all resources should be deallocated explicitly and implement thus IDisposable. Therefore, the Dispose method can decide if a resource is actually allowed to be destroyed.

This ultimately means that, while I need a thin resource management in the renderer, I do not need resource handles in that place. However, I will likely use a direct handle-based resource system later on top, where a single resource might compromise one or more renderer resource and additional data.

]]>https://wumpfblog.wordpress.com/2015/11/28/about-resource-handles/feed/0wuempftlbruempftlRelocating to Dublin experienceshttps://wumpfblog.wordpress.com/2015/11/14/relocating-to-dublin-experiences/
https://wumpfblog.wordpress.com/2015/11/14/relocating-to-dublin-experiences/#respondSat, 14 Nov 2015 20:49:35 +0000http://wumpfblog.wordpress.com/?p=32Continue reading Relocating to Dublin experiences]]>This blogpost is a modified version of the guide/report I wrote about relocating to Dublin for my company’s internal wiki (I stripped all sensitive or too specific information and cleaned it up a bit). There are a few co-workers which were quite happy about it, so maybe it helps somebody out there in the wild as well

Introduction & Background

This page sums up a few of the things I have learned during my relocation from Germany to Dublin. Everything here is based on personal experience, so take it with a grain of salt! Needles to say that I am expressing here a lot of biased and not always well supported opinions.

It is my first time to go somewhere abroad for more than a month. Before I was living in a student accommodation, so I had a very easy and cheap live

Accommodation

“The Hunt” – How to find an apartment

It is called “hunt” for a reason ;). Note, that I only looked for lettings. The search took me a bit over 2 weeks. During that time I increased my search intensity steadily.

Since I am now slowly getting settled down again after moving to Dublin, I have recently been thinking about starting a new side project. Doing side projects is a very natural hobby for me that allows me to play freely with ideas and gives me that cosy feeling of steady progress.
First, I thought about continuing with global illumination research (which I did in my master’s thesis) or helping out my friends at the amazing ezEngine. However, I realized that it might be fun to experiment with general infrastructure myself – something that I haven’t done for a long time. While the past showed me that I tend not to reuse code directly, there are many pieces of code in my archive that get recycled and improved in other projects – so it could be also useful for spin-offs etc.
My job involves mainly C++ coding atm, so I would rather use something else: I was tempted to use the Rust language but I am just too much in love with IDEs to try a large project with a fancy new language (note to myself: Blog about experiences with Rust). Instead I decided to go with an old friend of mine: C#

C# Pro/Cons

Personally, the biggest pros are the great IDE support (ReSharper makes it even better) and the lush standard library. Practically all engines I have seen so far have a huge Core of basic functionality for containers, threads/tasks, SIMD math, file access and so on. In .net there is already a lot of this stuff; 4.6 has even basic SIMD support, a work stealing task scheduler and the async functionalities are just a amazing! Don’t take me wrong, writing all this by hand can be a joy on its own, but I am just not in the mood for it ;).
Also, I have to admit that I just love some of the syntactical sugar of C#, e.g. properties, lambdas, events, …

On the other hand the garbage collector can strike at any time and can make it difficult to manage resource & lifetimes by hand where it is necessary (textures, buffers, but also high-level game resource management). Another con from my point of view is that I already now quite a bit about C#/.Net stuff which makes the whole matter a bit less interesting. On the other hand, recent versions introduced some cool stuff. Cross-Platform support is getting with Microsoft’s more recent open source initiatives really good, but I am still sceptical.

Things to try

A loose list of a few interesting things to which I look especially forward to investigate and experiment with during this project:

Modern multi-threaded renderer abstraction

Support for DX12, Vulkan but also OpenGL 4.5 (to prove the concept and make it work on my laptop)

might be necessary to keep the lowest abstraction level relatively high

Molecule Matters has a great series about this topic with many super interesting ideas. Can’t wait to try all this in C#

Resource system/manager

Component Entity systems

Especially interesting in the context of .net reflection stuff

Unit-testing in .net

Modern Scenegraph

Run on Linux without Mono

Make a WinRT application

…

Of course there will be also a lot of seemingly less exiting things that still might be fun, but to which I haven’t given much thought yet. For example logging, messaging, special data structures, etc.

Next, I will have a look at existing engines and start then experimenting & planning with the general project structure. While .net offers already a lot of stuff I will probably still need some kind of base library. As always, I will make everything public on GitHub.

To be continued!

]]>https://wumpfblog.wordpress.com/2015/11/01/new-sideproject-c-engine/feed/0wuempftlbruempftlDirectX12 Overview Diagramhttps://wumpfblog.wordpress.com/2015/09/11/6/
https://wumpfblog.wordpress.com/2015/09/11/6/#respondFri, 11 Sep 2015 10:00:33 +0000http://wumpfblog.wordpress.com/?p=6I made an overview diagram of the most important parts of Direct3D12 and how they work together. Most notably missing are Bundles. Making this diagram helped me to get an better overview and how things are interconnected.

If you find any mistakes or have ideas to enhance it I will gladly improve it .

The last few months I was pretty busy with my master’s thesis, which was a OpenGL 4.5 based project. Now that I’m on free foot again, it is time to try something new. I have some experience in D3D11 but its been a while.

Since there are already a lot of useful resources, I won’t write a full “First Triangle” tutorial but rather link to others. This post just sums up my first steps & insights. Maybe its useful for someone else too.

Pitfalls

Porting to Direct3D12 might be hard and does not guarantee that your application will run faster. Aras gave recently a talk about the issues you might encounter (siggraph15). Accordingly, its like porting from loose constants to constant buffers – you need a lot more “global” information about your rendering process.