C#, Azure, Xamarin, Web Services, Unity, Mixed Reality

Category: Mixed Reality

HoloLens a new approach

In the past few years, Microsoft has made many changes. One of the products they invested in is the HoloLens (known as Project Baraboo), which brings a new era in technology. The HoloLens device is a pair of smartglass developed and built by Microsoft. It runs on Windows 10 operating system and has a set of sensors that bring cutting edge technology in your space. Currently, it’s a little expensive. The development edition costs $ 3,000 and the commercial suite for business costs $ 5000. As Microsoft announced, the new HoloLens is expected be released with a great reduction in price and a larger field of view in the first quarter of 2019.

My Holographic experience

I had the opportunity to work on HoloLens device in the last year and it was really great. It’s based on the use of two cutting edge technologies, spatial mapping and spatial sound. Spatial mapping merges the real with the virtual world and as a result all the holograms seem real, while Spatial sound provides a 3D sound simulation experience using direction, distance and environment simulations. I can say that I have encountered some difficulties as HoloLens is a relatively new tool and many things had to be created from scratch, although the Microsoft libraries from Mixed Reality Toolkit (known as HoloToolkit) were very helpful.

In this tutorial I describe how to save and load data from text and XML files. The data I store is three-dimensional points into the mixed reality environment in which you can insert to list with gesture event air tap.

Create three GameObjects with Box Collider to use as buttons. With “Start Recording” tile you can start recording new three-dimensional points in the space with the use of air tap, with “Save Points” tile you can save the points in the text and XML file, with “Load Points” you can load the points from text and XML files. These GameObjects have a script for the OnSelect event.

Microsoft HoloLens has a variety of sensors, making it a very powerful tool. The device is placed on the head and you can view the space from their point of view. This guide represents how you can take a photo via HoloLens and project it on a holographic surface and how you can record and play videos.

Initially, create a new scene with the name “Capture-Play-Photo-Video”. It is important to set the Camera of the scene, the projection to Perspective. Also, you need to be in the right position as shown in the picture. Position (0, -0.02, 0), Rotation (0, 0, 0), Scale (1, 1, 1). Most importantly, the point where you choose to display the camera is the point based on which the other objects in space will be initialized.

Add two new GameObjects (Cube) in the unity scene under the names “DisplayPhoto” and “PlayVideo” and give the following transform to the images.Read More »

In the last few years there has been great advancement in the field of neural networks and the Artificial Intelligence (AI). Microsoft has also developed a set of Cognitive Services that are a set on machine learning algorithms to solve problems in the field of Artificial Intelligence (AI). The Cognitive Services APIs are grouped into five categories:

Vision—analyze images and videos for content and other useful information.

Speech—tools to improve speech recognition and identify the speaker.

Language—understanding sentences and intent rather than just words.

Knowledge—tracks down research from scientific journals for you.

Search—applies machine learning to web searches.

In this tutorial, we see how identification is used with the FACE API. There is also an additional GitHub guide for character recognition through the Computer Vision API (see the end of the article).

It is important to set in the Camera of the scene the projection to Perspective and to be in the right position as shown in the picture. Position (0, -0.02, 0), Rotation (0, 0, 0), Scale (1, 1, 1). Essentially the point where you choose to display the camera is the point based on which the other objects in space will be initialized.

You will need two GameObjects, the first GameObject named “PhotoCaptureManager” will contain the scripts and will be responsible for capturing the photo from the HoloLens device and then for sending to Azure Cognitive service to process it. The second GameObject named “AudioSource” will contain audio Source components in order to play the voice from text-to-speech implementation. For example “He is pretty much 26 years old and looks happy”.

Before you start coding, you need to create an Azure account and save the service key in order to use in your code. Choose the billing package. With F0 free package you can create 20 calls a minute and 30k a month.

This guide includes two scenes. In the first scene, you have to calculate the distance between HoloLens and any surface from the real world and the second one is a calculation which summarizes the distance from multiple three-dimensional points.

The calculation of the distance between HoloLens and any three-dimensional point in the space is described by the following method. You can use the spatial mapping component from HoloToolkit in order to recognize the surfaces from the real-world environment. These two points are the location of the HoloLens and the position that HoloLens recognizes as a surface through Gaze.

Gaze is a primary form of targeting within mixed reality. With the gaze cursor the user is looking in the real world (if you use spatial mapping) and in the holograms and with the use of gestures they can interact with their environment. The primary way for a user to take action are via gestures, motion controllers and voice. The main gestures are air tap, double air tap, hold, drag and you can create your own gestures.

In this guide you can create a HoloLens application from scratch in order to learn to use gaze cursor and some gestures. For the development of the application we use the unity platform.

Create a new unity project and into the project new scene with the name “GazeGesture”. Initially, you have to make some adjustments in order to use and build successfully the HoloLens application. Choose Edit -> Project Settings -> Player. On the XR Settings tab, check the Virtual Reality Supported checkbox and add “Windows Virtual Reality” to list. To extract the project, go to File -> Build Settings …, there choose the Universal Window Platform tab and set Device to HoloLens and Build Type to D3D.

It is important to set in the Camera of the scene the projection to Perspective and to be in the right position as shown in the picture. Position (0, -0.02, 0), Rotation (0, 0, 0), Scale (1, 1, 1). Essentially, the point where you choose to display the camera is the point based on this which the other objects in space will be initialized.

To create the cursor, you have to add a new GameObject with the name “GlobalCursor” with Scale (1, 0.5, 1) and as a child of this GameObject another GameObject with a name “GlobalCursorMesh” which must include the components Mesh and MeshRenderer. You can choose Mesh from Microsoft WindowsMixedReality Toolkit.