This blog attempts to be a collection of how-to examples in the Microsoft software stack - things that may take forever to find out, especially for the beginner. I see it as my way to return something to the Microsoft community in exchange for what I learned from it.

29 June 2016

I am currently finishing up a nice little demo app for the HoloLens but ran into a weird problem. Originally the app was called ‘Gazer’, as I only wanted to test Gaze Input, but it turned out to become a lot more. So I wanted to call it “CubeBouncer”.

It’s not so hard to do, you got File/Build settings, then go to “Player Settings”, and in the inspector on the right hand side you see a Store logo. Click that, select the Icon tab, and under “Short name” you will see the name that will be displayed on the HoloLens app tile. You can also select on what types of tiles it needs to appear. It quite reflects the Visual Studio UWP Manifest editor, which is logical, as this is what the manifest is generated from.

Build the app by hitting the Build button as usual when you have made changes in Unity, Visual Studio will say the project has changed and prompts to reload it, you build and deploy it to the HoloLens or the emulator, pin your app to the start screen – and you will see nothing has changed. The old name is still displayed.

Turns out that Unity UWP generation is a bit too clever when it comes to generating the UWP – apparently it updates not everything, but only the things that have changed. And I think it fails to take changes into the manifest into account.

The solution is very simple: don’t overwrite the generated UWP app. Close Visual Studio, delete the generated UWP app entirely (warning – only the generated app code, not your entire Unity project), only then hit Build in Unity (you will notice that takes quite a bit longer, as it needs to restore NuGet packages and stuff too), open Visual Studio again and then if you deploy your app to HoloLens, the short name will have changed.

Possibly there are smarter ways to do this, like only deleting the Manifest file – I have not tried this – but this is a sure fire way to fix this. It’s a bit cumbersome, but changing the name of an app or a tile isn’t exactly something you do ten times a day, so it’s not that much of a problem. But it’s a nice gotcha, so I thought it best to document it.

11 June 2016

Picking the challenge apart

An Italian proverb says that a fool can ask more questions than seven wise men can answer – a modern variant could be that a designer can think up more things than a developer can build. I don’t pretend to be the proverbial wise man, and neither do I want to call my designer colleague a fool, and when he came up with the idea of introducing text balloons with context relevant information, floating on top on the rest of the UI, cross-platform, and preferably appearing with a nice animation, I indeed had to do some head scratching.

What he meant was this, and this is exactly how I created it

The issues I had to tackle, were:

How do I create a text balloon in the first place, with a kind of pointy bit pointing to the UI element it belongs to?

How do I get the absolute position of a UI element - that is, the one the user taps?

How do I show the text balloon in situ?

Text balloon 101

This text balloon consists of a translucent grid that ties the components together. It contains two grids, one of them containing the label. The first grid is the bit that points up. This actually is a 15 by 15 square, rotated 45⁰, and moved a little bit to the left using the new Margins property that has finally made it to Xamarin Forms. The second and biggest grid is the green rectangle actually containing the text you want to show. Because it’s in XAML after the square, drawing precedence rules make that it be drawn on top of the first one. You would not see it at all, if is wasn’t for the fact this grid also has a margin - of 7 on the top so about half of the rotated square. The net result, as you can see, is a triangle sticking out of the rectangle, making the optical illusion of a kind of text balloon. In XAML, this looks like this

The complete text balloon is contained in the “MessageGrid” grid; the pointy bit upward is emphasized using red and underlining. The complete control is contained within yet another grid “MessageGridContainer”, that fills the whole screen – or at least the part in which the text balloons appear. Is has three functions:

It provides a canvas to place the actual text balloons on

It is an event catcher – as the user taps ‘anywhere on the screen’ the text balloon disappears

It makes sure the text balloon never gets wider than 75% of the screen (this was a designer requirement) – hence the columns.

Some important details to take note of:

A few elements have set the property “InputTransparent” to true. This means they will never receive any events (like tap) but those will be received by the elements lying ‘below’ them. In other words, they don’t block events. This makes the text balloon disappear even when you tap on the text balloon itself, as the events goes downwards and is processed by MessageGridContainer

MessageGridContainer itself is not opaque but has BackgroundColor "#01000000", that is, 1% black. For all intents and purposes it is opaque in the sense that you don’t see it, but if you leave it totally opaque is will also be opaque to events - on Windows 10 UWP. A little concession to a cross platform issue.

This whole contraption is called “FloatingPopupControl” – this is the control that handles showing, displaying and eventually removing the text balloon. ‘Something’ has to call it’s ShowMessageFor method to tell it what the balloon should contain, and under which control it should appear.We will come to that later

Determining absolute position of the ‘anchor element’

The anchor-element is the element under which the text balloon appear should appear when it’s tapped – in this sample, the i-symbol. It is actually pretty to simple to find the absolute position of a relatively placed element: this is achieved by going recursively upwards via the “Parent” propertie and get the sum of all X values and the sum of all Y values. You can actually find hints to this in the Xamarin developer forumsin the Xamarin developer forums, and I have put this into the following extension method:

Positioning, showing, animating and removing text balloons

If you look at the code in the ShowMessageFor method - in the FloatingPopupControl code behind – you’ll see the code is only deferring to FloatingPopupDisplayStrategy. This is done because it’s not wise to put much code into a user control if you want to re-use part of that intelligence easily. It also makes adapting and changing animations easier. FloatingPopupDisplayStrategy has the following constructor:

Note: the delta uses in the calculation is a value intended to use as a correction value, in case the standard calculation does not yield the desired result (i.e. location). This will be explained later on.

And finally, the user must be able to dismiss the text balloon. This is done using the ResetControl methods. As we have seen in de constructor, this method gets called in case the user types at the invisible canvas, or if the canvas’ size changes.

This method does not need to be called explicitly at initialization, since he invisible grid changes size at the start of the app (because it gets child elements – the MessageGrid and its children), and the event wiring makes this call happen anyway. Another important reason to attach this method to the SizeChanged event is that in Windows 10 UWP apps windows sizes actually can be changed by the user. This may cause text balloons ending up in what is no longer being the right place, so they need to be removed as well. After all, as long as the text balloon is visible, the invisible background blocks any input, so as soon as the user starts working with the app, in any way, the text balloon needs to disappear and the app needs to be ready again.

Behavior intercepting tap event and relay to control

The only thing missing now is something to get the whole process going – respond to the tap on the i-symbol, providing the text balloon contents, and provide some optional positioning correcting for the text balloon. This is done by FloatingPopupBehavior:

This behavior is actually rather simple – as soon as the control tot which is attached is tapped, it calls the ShowMessageFor method of the control referenced in de PopupControl property. There are three additional property for determining which text is actually displayed, and two optional properties for a delta X and delta Y which, as we have seen, are included by the control when it actually places the text balloon on the right place.

In red the actual i-symbol with the behavior attached to it, in green the popup control (including the label) itself. In the behavior’s properties we actually specify the text to be displayed as well the PopupControl reference, indicating this is the UI control that should actually handle the displaying of the text balloon. In addition it sports an optional extra delta x and delta y. Of course this could be hard coded into the control, but to have this extra flexibility in design time makes for an easier ‘constructable ’UI. As you can see, as soon as the parts are in place, actually using and re-using the components is pretty easy, making adding floating text balloons with contextual relevant information very easy indeed.

Also notice a neat trick to make sure that especially nice in Android, that sports a great rage of resolutions. I took an intentionally too big picture for the i-symbol, which is automatically sized to the height of the entry by using HeightRequest="{Binding Height, Source={x:Reference NameEntry}}"

Some consequences of this approach

As stated repeatedly, a (nearly) invisible grid covers the whole screen, or at least part of the screen, while a text balloon is displayed – to give the text balloon space to be placed in, and to intercept a tap so it will be removed as soon as the user starts interacting with the app. The flip side is that the app is effectively blocked until the user taps, and this tap will not do anything but removing the text balloon. A plainly visible button will not respond while the text balloon is visible – that requires yet another tap. This may seem annoying, but I don’t think this will put off the user in any significant amount, as it;s likely he will stop using this functionality pretty soon as he/she has gotten the hang of the app. This is only an onboarding/adopting thing. You read the car’s manual only once (if you do it at all, and then never again unless in very extraordinary circustances)

Conclusion

Using only some pretty basic means, a few nifty tricks and a clear architectural approach it appears to be pretty simple to build a kind of re-usable infrastructure enabling the fast and flexible addition of context relevant information, which is displayed in a visually attractive way. It’s very easy to add text balloons this way, and it’s a useful tool to make onboarding and adoption of an app easier.

As usual, a sample project containing this (and previous code) can be found on GitHub

01 June 2016

After following some on-line video’s and snooping through some other’s people code I felt I had to start bottom-up to get at least a feeling for how you setup a HoloLens app – even though I don’t have a HoloLens. Call it my natural curiosity. So I followed my fellow MVP Morten Nielsen’s blog – highly recommended – as he starts with the real basics. Very good if you hardly have a clue what you are doing ;)

In his third post, I ran into a snag. According to his second post, I had unzipped the HoloLens toolkit into the assets folder as instructed, and started to add occlusion to my project. Morten explains in detail what this is – it makes holograms disappear behind physical objects like a wall when they are ‘in front’ of the holograms (which, in reality, they never can be as they are projected on a screen not 2 inch from your eyes, but that is the magic of HoloLens).

So I added a rectangular box as a Hologram, having it stick out of a wall like halfway, so I was supposed only to see the front part. That I did, but I did also see something else:

Yikes – that is my box all right, but where did that horrible magenta color come from? To this moment, I still don’t know, but I found out how to fix it.

First of all, it really helps if, in addition to making the settings to your project that Morten describes in his first post, you tick at least “Unity C# project” as well in Build Settings:

I do the “Development Build” as well, although I don’t really know if this is neccesary).

This makes all the scripts end up in your Visual Studio solution as well, and what is more, you can debug them. Thus I learned that the script “SpatialMappingRenderer.cs” (in folder Assets\HoloToolkit\SpatialMapping\Scripts\SpatialMappingComponent) looks for materials “Occlusion” and “WireFrame” that are clearly not there.

If you debug this script (you can now thanks to the setting above) you will see both OcclusionMaterial and RenderingMaterial end up as null. The materials are in HoloToolkit/SpatialMapping/Materials but changing “HoloToolkit” to “HoloToolkit/SpatialMapping/Materials” does not have any effect.

So I went back to the Unity editor, selected the Spatial Mapping Renderer script again, changed the dropdown “Render mode” from “Occlusion” to “Material”, and that made the property “Render Material” pop up. I found the Occlusion Material on top of it, and it was accepted.

If I now debug the code, the Occlusion material is still null, but RenderingMaterial is set, and lo and behold:

Now I am seeing what I expect to see – a box sticking out of a wall.

Disclaimer – I am just happily stumbling along trough Unity3D and HoloLens, not hampered by much knowledge of either. I found a problem (that I maybe caused myself) and I fixed it. I hope it helps someone. If I messed up, maybe someone can enlighten me. Unlike some other people, I am not afraid to look a n00b because in this case, that’s just what I am.

The project itself can be found in a good old-fashioned ZIP file as I don’t want to clutter up my GitHub account with every little thing I try at this stage. I am still in the stage of ‘programming by changing things and observing their effects’, so I hope you will humor me. And maybe this will help someone, who knows.

Feedback, comments and tokens of appreciation

If you spot things that are incorrect, or if you don't understand what I mean, please drop a comment on the offending article and I will help you ASAP. You can e-mail me at joostvanschaik at outlook dot com or contact me via twitter.

If you find the information on this blog useful (and apparently some 600+ people per day do on average) please let me know as well, that encourages me to keep doing this. Or do tell others - that made me an MVP; who knows what more it might bring ;-P

Disclaimer and legal stuff

Although I take great care in providing quality samples, all postings, articles and/or files on this site are provided "AS IS" with no warranties, and confer no rights. The views expressed on this blog are strictly my own and do not necessarily reflect the views of my employer, or anyone else on the planet for that matter.

I usually make original content, sometimes building upon other people's work. Sometimes I explain things that can be found elsewhere because I felt what I read was not clear enough for my limited mind so I explain it the way it finally clicked with me. In all cases I take great pains to be sure to link to people or articles who deserve the credit. If you think I have shortchanged you on the credits please let me know.

Please note, I do not work for Microsoft and while I proudly wear the title of "Microsoft Most Valueable Professional", my opinions, files offered, etc. do not represent, are approved of, endorsed by or paid for by Microsoft. The only power behind it is me and my sometimes runaway passion for parts of Microsoft's technology.