Getting It Just Right – Benchmarking On HoloLens

Finding the exact balance in any recipe is work that requires a fine taste and a lot of hours invested in the kitchen. A bit too much of anything is almost as bad as being a wee short. Never too much baking, but neither too little. The perfect equilibrium is what we aim for at Umbra, performance versus quality. Not an easy job, but those are boring anyway.

There’s a lot going on behind many augmented/mixed reality applications: spatial recognition, gesture input, network communication, etc. And as an autonomous computer, HoloLens must deal with all of that in real time. Balancing the performance is a crucial issue because of the extra limitations that the hardware imposes compared to desktop platforms. Sometimes you just need the right amount of details, as we don’t want to squeeze all the juice within a couple of minutes. These extra challenges that make it harder to find that fine line where power, performance and quality meets perfectly just feels good.

We are working hard to get HoloLens support for our upcoming product ready. And although it is always exciting to work on new platforms like this, they come with their fair share of new questions and issues. But don’t worry my friends, Umbra is here to save the day – or at least to save you some headaches while loading humongous CAD models into your favorite holographic device. So, I crafted some tests that will tell us how Umbra actually helps a lot to get that performance where it should be.

The Tests

For this first benchmarking we did quite simple tests that were designed to assess performance rendering geometry with different volumes of triangles. A Unity application with a single scene is run on the target device, and the HoloLens Device Portal then provides us the resulting performance graphs. The test scene contains one 3D model that is displayed right in front of the camera. All the elements in the scene are static (even the headset camera) and the only light source is ambient light. Pretty minimal, but then again we just want to know how many triangles we can render in real time.

Each model is tested with different levels of optimization: none (the original geometry), average and aggressive. After that the we employ three different shaders for rendering on each of the models: Standard (PBR default shader in Unity), StandardFast (an optimized version of Unity’s standard supplied with HoloToolkit) and an unlit shader.

Recommended settings

It’s very important to set the right graphic options in order to get a smooth performance – check these recommendations from HoloLens development site for more details. Also depending on the content of your app, remember to set the clipping plane of your camera accordingly.

The Results

The following graphs show the frame rate for the battery of tests: with two different 3D models, three levels of optimization and using the previously mentioned shaders for rendering.

Models below 200.000 triangles can run smoothly at 30 FPS, not perfect but valid for some applications.

HoloLens is comfortable rendering 80K triangles simultaneously. Around this value it’s important to craft your shaders carefully because heavy calculation in the fragment shader will have a severe impact on performance.

There’s no big impact using StandardFast (HoloToolkit) over the standard PBS shader provided by Unity

As expected, unoptimized high poly models run on extremely low frame rates. As soon as we reduce the number of triangles below 200K we can see that performance begins to rise. When the number of polygons descends to 100K we can see how the different shaders have a deeper impact on the frame rate.

Another thing I discovered during the benchmarking is that enabling Mixed Reality Streaming (you can stream everything you see through HoloLens to a computer, for example), the scene automatically drops the frame ratio to 30Hz, despite the quality of the video selected for the stream. This is a limitation introduced to ensure a minimum quality on the stream.

The Conclusions

It’s clear that with Umbra technology it is indeed possible to bring complex models ( like large landscapes or detailed architectural models) to Microsoft’s HoloLens without crunching its GPU and CPU. Besides this benchmarking, the data will help us to fine tune Umbra’s support and evaluate future improvements as well. These test models were from the amazing people at Spotscale.