Going Cross-Reality with Unity

When we started working on Pryzm, our application for viewing 3d content stored in our cloud, we faced a bit of a dilemma. Umbra has over a decade's worth of experience developing middleware for a dizzying array of platforms, but not nearly as much experience in developing multi-platform user-facing applications.

We wanted to develop a 3D viewer app which you could use on mobile, on the Hololens, in VR, and embed within websites. Since we live in the golden age of cross-platform application development right now, we wouldn't have to write everything from scratch ourselves, no matter how much fun we, as developers, would have in doing so. The only question was: which technology should we choose?

One of the initial ideas was to use Javascript and WebGL. Embedding within websites is trivial, and the HoloJS framework allows you to easily port over your WebGL app to the Hololens. As amazingly diverse as web technology based applications are, we decided to opt for Unity -- which arguably makes deployment onto AR, VR, and mobile devices simpler than a web renderer. Building Pryzm on top of Unity would mean that we could use parts of our own Composit for Unity plugin, which is a good way of figuring what works well and what doesn't.

Cross-platform LOD and streaming: to C# or not to C#?

Our viewer has to do quite a bit of work: it has to figure out what parts of the scene are visible, and what level of detail they should be shown in, and then stream in the required content asynchronously. From the get-go, we implemented the visibility and level of detail calculations as a native plugin to Unity. This means that we included a library written in C++ with our Unity project, and called it from C#. Each platform has its own version of the library, and we tell Unity which library to use during a build via the library's metafile. This means that, for instance, if a platform does not have its own library in our Composit Plugins folder, then the Unity build for that platform is not going to succeed.

We used Unity for loading the content from the cloud. Naturally, loading something at runtime means that all the work has to be done in a separate thread. C# abstracts away the platform that you're on, so streaming may seem like a natural thing to implement in C#. Unfortunately, there exists a tradeoff. Unity targets different .NET versions depending on the platform. Hololens uses the .NET compact framework, which doesn't expose threads like the regular .NET 3.5 framework does on other platforms. We ended up conditionally compiling a bunch of code against different API versions. This is exactly what platform-specific C++ is like!

It made more sense to handle everything in the native plugin. We already had all the platform-specific C++ code we needed anyway, thanks to our middleware development history. This also presented a very nice separation of responsibility between the Pryzm application and the native plugin. The Pryzm application simply connects the plugin to the desired scene, and then the plugin gives the application 3D assets to render, based on the application's camera.

Composit for Unity is essentially a Unity layer over the aforementioned native plugin. By simply including the Umbra GameObject in your Unity scene and pointing it to the desired view, level of detail based streaming is available in your Unity app as well.

It should be mentioned that implementing our streaming logic in a native plugin does not conflict with our goal of building a web viewer. Unity can actually compile C# into Javascript by first compiling C# into C++, and then using the Emscripten compiler to compile the C++ into Javascript. Our native plugin gets fed into Emscripten and is converted into Javascript along with the rest of the Unity project.

One UI to rule them all

We needed to build a simple UI in order to allow users to select the view they wanted to see within the app. Luckily, Unity's built-in canvas UI system allows you to build any sort of UI you could imagine. Here's the part where the platforms we are targeting make things interesting: for Hololens AR and VR, your UI has to be spatial -- it has to float around in front of you. The way you point and click can happen via a gaze pointer and hand gesture, or via a controller. On mobile, you naturally have to build a screen space UI. Our web viewer doesn't have a UI at all because it gets its input from the surrounding web page! Each of these platforms needs a completely separate system of handling input.

Initially, we tried to build a single scene which could handle all these disparate platforms. We wrote our own input module which could handle mouse clicks, as well as gaze, hand gestures, and VR controller pointers so that we could use the same canvas UI for any platform.

Unfortunately, once you start adding support for the web and mobile into a scene containing AR support, you end up with a jungle of UI object hierarchies which all need to be activated and deactivated depending on the platform. More importantly, once you introduce iOS dependencies into a scene containing Hololens-specific code, your scene will compile for neither platform. So, we split our Pryzm app up into multiple scenes -- one per platform.

No more headaches

Our Pryzm apps, as they currently stand, are thin wrappers around our native plugin. They contain a game object for our runtime, as well as an object hierarchy for the UI. In this way, starting work on a new platform is faster than ever -- we can ignore what the other platforms are doing, and our runtime, being merely an interface to a plugin, simply works regardless of .NET API version.

Want to see for yourself? Go to umbra.io to sign up and get the Composit for Unity plugin!

Learn more about AR's potential here, and for more information on 3D content in unity check out this post.