How to Use Machine Learning to Show Predictions in Augmented Reality – Part 2

Part 1 Recap

This tutorial is part of a multi-part series, in the first part, we loaded our data from a .csv file and used Linear Regression in order to predict the number of patients that the hospital is expected to receive in future years.

Tutorial Source Code

Improve our UI

A scene with the menu, where the user can predict the affluence of patients in the next years, simply using an input field and the ability to read the data from the .csv

A scene where we will show the data visualization, where the user will have the possibility to look at a 3D histogram of the data and the prediction. Furthermore, we will allow the user to customize the data visualization, increasing and decreasing the size of the histogram.

First, we create a new button inside the canvas called “GoToGraphScene”, later we will reorganize the UI, this button will allow the user to navigate through the scenes.

Here we create a new script called NavigateScenes.cs in the Scripts folder, where we load the new scene and allow the user to change the scene.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

usingSystem.Collections;

usingSystem.Collections.Generic;

usingUnityEngine;

usingUnityEngine.SceneManagement;

publicclassNavigateScenes:MonoBehaviour{

publicvoidLoadScene()

{

Scene scene=SceneManager.GetActiveScene();

SceneManager.LoadScene("tutorial_graph");

}

}

Note that I included “using UnityEngine.SceneManagement”.

We need to execute the script when the “GoToMenu” button is clicked by the user, in order to do that :

Add the script to the “GoToMenu” button

Click on the plus sign on the On Click section

Drag the button itself on the empty field

Select from the dropdown menu in the On Click section the NavigateScenes, then LoadScenefunction.

We create a new scene called “tutorial_graph” in the assetfolder.

Save the changes in the “tutorial” scene and now let’s repeat the same process for the navigation button in the “tutorial_graph” scene:

Create a canvas

Create a panel inside the canvas

Create a button called “GoToMenu” inside the canvas,in order to go back to the menu

Anchor the button on the top and centre of the canvas

Assign the script called “NavigateScene.cs” to the button

Execute the script when the button is clicked

Save changes

We can optimize this process, instead of creating a new script for each scene, we can add some line of codes in the”NavigateScene.cs” script and load the different scenes based on which is the active scene at the moment, you can find more details about this topic in the SceneManager docs https://docs.unity3d.com/ScriptReference/SceneManagement.SceneManager.html. We use an if because we have only two scenes, with more scene would be better a switch statement or a more complex architecture, but for now, if we are in “tutorial” we will go to “tutorial_graph” or the inverse operation, so a simple if it’s enough.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

usingSystem.Collections;

usingSystem.Collections.Generic;

usingUnityEngine;

usingUnityEngine.SceneManagement;

publicclassNavigateScenes:MonoBehaviour{

publicvoidLoadScene()

{

Scene scene=SceneManager.GetActiveScene();

if(scene.name!="tutorial")

{

SceneManager.LoadScene("tutorial");

}

else

{

SceneManager.LoadScene("tutorial_graph");

}

}

}

This is the look of the new scene, feels empty, but we will add awesome things later in the tutorial.

Open the build settings from the File menu, we need to add in the build window both the scenes, we can do it simply dragging them into this window, after that our build settings window should look like this:

Now, clicking on the button should change the scene in both of the scenes, so the user can now navigate successfully from the menu to the data visualization scene and back.

Let’s go back to the first “tutorial” scene to improve the UI.

Would be great to fold the dataset text with a button, so we create a new button called “ShowData”, remember to create this element inside the canvas.

Inside that button, we create a Scroll View from the UI menu, the scroll view object allows the user to scroll through a long list of data. I disabled the horizontal scroll, anchor the scroll view at the bottom of the button, as you can see in the screenshot.

Set the value of the Scrollbar Vertical Scrollbar script to 1, so the scrollbar starts from the top, it is important to set this value, otherwise, the text will start in the middle of the scroll view.

Now we drag the DatasetTextobject inside the Content object of the scroll view, remember to anchor this element (I used the top left anchor), so even in with different screen sizes the position will be relatively the same.

A long list can be tedious to watch, so it is better to make the user choose to expand it or not. In order to add the show or hide functionality of our “ShowData” button, we create a script called “ShowHide.cs” in the Scripts folder.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

usingSystem.Collections;

usingSystem.Collections.Generic;

usingUnityEngine;

publicclassShowHide:MonoBehaviour

{

publicGameObject showHideObj;

publicvoidShowHideTask()

{

if(showHideObj.activeSelf)

{

showHideObj.SetActive(false);

}

else

{

showHideObj.SetActive(true);

}

}

}

As you can see in the code, using an If we check if the object is active or not, so we hide or show our Scroll View. Feel free to reuse this code to add more functionalities like this in other parts of the UI.

Add that script to the “ShowData” button and assign the ScrollView as the target object.

Next, we add the show/hide functionality on the click event, just dragging the button itself in the On Click() section, selecting the ShowHideTask() from the “ShowHide.cs” script.

Disable the Scroll View element.

Now, clicking on the button should hide or show the dataset list.

Let’s centre the UI elements like in this screenshot, all elements are anchored at the top center, this will make our application easier to use on smaller screens, such as small smartphone devices or low-resolution tablets.

Load the data from the script “LinearRegression.cs” and using a for loop we pass the data inside the “createBar”function.

Set a parent container.

The “createBar”function creates a primitive cube inside the parent container, so if we want to clear the graph we simply remove the children of the parent container. The cube has a fixed width, its height is taken from the number of patients in a particular year. We use the scaleHeight in order to modify the size of the height of the visualization, offset helps us to have a distance between the bars.

There a scaleSize value that we will use later to allow the user to customize the graph.

There is an offset value that allows us to create a bin next to the other.

This part of the code:

1

((float)LinearRegression.quantityValues[i]/scaleSize)/2

assures that each bar of the histogram will start from the same baseline, otherwise, the bars will be centred vertically.

This line:

1

cube.transform.position=newVector3(Padding_x,Padding_y-100,400);

contains fixed values in order to position the histogram in front of the camera, feel free to change this values (100 is y, 400 is z) in order to position the histogram where you like. In the third part of the tutorial, we will use a target image and the histogram will spawn on the target, so those coordinates will be removed.

We need to assign the “GenDataViz.cs” to the GraphContainer object, in the graphContainerfield of the script we drag the GraphContainerobject.

Now, everything should be fine, let’s start the game scene from the “tutorial” scene and navigate to the “graph_tutorial” scene using the go-to button, the 3D histogram should appear near the centre of the screen.

The histogram looks really boring and it’s hard to distinguish one bar from another, let’s add some colors, in this snippet of code we add a random color to each bar.

Now, let’s add more customization to the graph, allowing the user to modify the size of the graph using two buttons, so we create two buttons called “IncreaseSize” and “DecreaseSize” inside the panel.

Remember to anchor them, I personally use the top and centre anchor.

We need to modify our “GenDataViz.cs” script in order to these functionalities, so we add the two functions in order to decrease or increase the scale size of the graph, increasing ScaleSize the graph would become smaller, decreasing ScaleSize the graph would become bigger. Feel free to change the ScaleFactor at your will.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

using UnityEngine.UI;// Required when Using UI elements.

publicclassGenDataViz:MonoBehaviour

{

intscaleSize=100;

intscaleSizeFactor=10;

publicGameObject graphContainer;

intbinDistance=13;

floatoffset=0;

// Use this for initialization

voidStart()

{

CreateGraph();

}

publicvoidClearChilds(Transform parent)

{

offset=0;

foreach(Transform childinparent)

{

Destroy(child.gameObject);

}

}

// Here we allow the use to increase and decrease the size of the data visualization

Now, if everything works correctly, the increase button should make the graph bigger and the other should decrease the size.

Et voila, now when the user changes the scale size input, the graph should be recreated.

Let’s add a reset function, so if the user gets lost in increasing or decreasing the size of the graph, the user can reset the size easily with a button. Then, we create a new button called “ResetSize” inside the canvas. Anchor this button to the top center.

Let’s add a function inside “GenDataViz.cs”, here we add the “ResetSize” function that simply reset the ScaleSize value and recreate the graph calling the “CreateGraph” function.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

usingSystem.Collections;

usingSystem.Collections.Generic;

usingUnityEngine;

usingUnityEngine.UI;// Required when Using UI elements.

publicclassGenDataViz:MonoBehaviour

{

intscaleSize=100;

intscaleSizeFactor=10;

publicGameObject graphContainer;

intbinDistance=13;

floatoffset=0;

// Use this for initialization

voidStart()

{

CreateGraph();

}

publicvoidClearChilds(Transform parent)

{

offset=0;

foreach(Transform child inparent)

{

Destroy(child.gameObject);

}

}

// Here we allow the use to increase and decrease the size of the data visualization

Now, something is still missing, it’s the bar of the prediction about how many patients the hospital will need to take care in a future year. We will add the prediction as a new a bar at the end of the histogram.

First, Let’s modify the “LinearRegression.cs”script, so we can call the “PredictionTask”function from the “graph_tutorial”scene and give the user the ability to generate different predictions from the new scene.

As you can see we added a new static variable, “PredictedOutput” and modified the “PredictionTask” function, so this function can take a value from an input field and output the value in the “PredictedOutput” variable.

We convert the prediction result to an int because we are predicting the number of patients so a float number would not be appropriate, we can’t have one patient and a half! 🙂

Now, let’s modify the “GenDataViz.cs”

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

usingSystem.Collections;

usingSystem.Collections.Generic;

usingUnityEngine;

usingUnityEngine.UI;// Required when Using UI elements.

publicclassGenDataViz:MonoBehaviour

{

intscaleSize=100;

intscaleSizeFactor=10;

publicGameObject graphContainer;

intbinDistance=13;

floatoffset=0;

// Use this for initialization

voidStart()

{

CreateGraph();

}

publicvoidClearChilds(Transform parent)

{

offset=0;

foreach(Transform child inparent)

{

Destroy(child.gameObject);

}

}

// Here we allow the use to increase and decrease the size of the data visualization

That’s all for this second part of the tutorial, I hope you have enjoyed it!

In the next and last part, we’ll write the code to visualize our 3D data visualization in Augmented Reality using EasyAR. In the meantime, have fun with our project, trying to customize the 3D data visualization and the UI as you prefer! 🙂

See you soon!

Published by

Elisa Romondia

Data Analyst and Blockchain Developer in the top 60 Forbes list worldwide and Italy Inspiring Fifty women in STEM. She has an MSc in Psychology and several years of experience with Unity and AR/VR Development. She realized the first ARKit app released in Italy and she was the first woman to win an official Italian national hackathon. In 2017 she founded Devoleum.com, an international awarded Blockchain and AI solution for the food supply chain. View all posts by Elisa Romondia