miércoles, 15 de febrero de 2017

SAP d-shop’s Virtual House – A journey from Physical to Virtual

Some
time ago…our good friends from SAP d-shop Newtown Square (Namely John Astill et
all) built a IoT House for SAP Insurance. This little house (hand made by the
way) used an Arduino Nano, a bunch of sensors and LED lights…and…which is
pretty cool by the way…a 3D Printed washing machine with a water sensor…and of
course…it was and it is…IoT enabled.

We thought it was pretty cool…so we have one at our own SAP
d-shop at Silicon Valley and it had become a key part in all our d-shop tours.

Then…some time later, our friends from HCP
Marketing (Namely Joe Binkley et all) and Intel build a Smart Building. A
really nice building…controlled by Amazon Alexa that used an Intel Galileo,
some Arduinos as well as servos, lights, a solar panel and even a
fan…everything again…IoT enabled…but also as you may have guessed…voice
controlled…so you can send the elevator up and down…open or closed the doors
and even send the whole building on emergency mode…gladly…we had keep it on the
d-shop for quite some time and it’s another of our “wow” factor demos every
time someone comes to visit…

Having these two available for us…slowly sparked the fire of
innovation and creativity…why don’t we build a Virtual House that can be used
on the Oculus Rift and it’s controlled by Alexa?

The architecture is pretty simple… The Heroku server is just
an echo server, so it will repeat everything we pass to it as a JSON response.
Our Unity app is constantly checking the Heroku server to see if there’s a
message to respond to. Of course, for this to work as intended, we need to
setup a skill on Amazon Alexa just to update the server. So, when we say “open
door”, then Alexa will send a command to the Heroku server and this server then
will produce an “open door” message in JSON. Our Unity app will read the Heroku
Server, and act accordingly by opening the door…of course, we don’t want this
to happened all over…so after Unity executes the action it sent a null message
to the Heroku server, so next time the JSON response is going to be null as
well and Unity will simply wait for the next valid command.

If you want to take a sneak peak of how the
Virtual House looks like…here are a couple of screenshots…but don’t forget to
watch the video, J You will get the full experience -;)

Now…this project started as a “Project in a Box” (for
internal only…sorry about that) which means…all the source code and
explanations on how to build it from the scratch should be provided…but…for
obvious reasons…that didn’t happened L So instead…we turned this into a
“Product in a Box” meaning that (Sorry again…internals only) you can download
the compiled application and simply edit the configuration file to have it
running on your own J No source code is provided by obviously a nice email can get
you that -;)

Now…that I got your full attention…please watch the video J It’s a nice journey from the IoT
House to the Virtual House passing by the Smart Building…

Now…you may wonder about the 3D Models used for this Virtual
House…as you can see on one of the images, most of them were downloaded but
some of them were developed in house J using Blender…Like the Amazon Echo,
the 3D Printed Robot and name tags, the Amazon Echo and obviously the house
itself J For some other things like the
plants and tables…those were imported into Blender and “hand painted” as the
textures were not available.

Now…something that we believe it’s pretty
important…is to list all the Pain Points and lessons learned while developing
this application…

Pain Points and lesson
learned:

As this is a Product in a Box and not a Project in a Box,
we’re not going to include the source code for this application, but what we’re
going to do instead is let you know the pain points and lessons learned that
came from this project.

Unity
uses the .NET Framework 3.5, which is already deprecated by .NET 4.0 so many
things are not going to work simply because they haven’t been implemented…and
why is that? Well…Unity uses Mono (which is .NET for Linux) and I guess they do
it to maintain uniformity in all platforms. While Mono remains on .NET 3.5,
Unity will not likely upgrade either.

When
loading scenes, the lighting gets all messed up…so you start in level one…more
to level two and suddenly it looks like nighttime…the solution to that is
simple…choose “Window à Lighting à Lightmaps”, uncheck the “Auto” checkbox and press
“Build” to bake the light again.

Coroutines
are simply awesome. Normally, you can’t make your application wait or sleep…but
by using Coroutines you certainly can…Coroutines are like threads.

When
using a light, make sure it’s turn off while the character is not in the room,
because this will save some graphic processing and because even virtually…we
need to be environment aware…

Unity
doesn’t have a wrap function or property for 3D Text…which is kind of problematic
especially if you want to do a Twitter Wall…so your only chance is to build you
own…although that’s not that hard…simply grab the incoming text, split it by
space into an array…concatenate each word by checking first if the length of
the string is lower than our threshold (which should be the maximum number of
characters that fit where our 3D text is), is the string is bigger than the
threshold, we simply add a carriage return (“\n”) before doing the
concatenation.

As
your application grows you might feel the need to duplicate some assets, which
is perfectly fine and doesn’t add too much processing (Especially if you create
a Prefab and use that prefab), but don’t forget to assign them unique names,
otherwise you’re going to have a headache if you application needs to interact
with those assets.

Sometimes
you will download some 3D models from the web…other times you will create them
using Blender…but don’t forget that sometimes just a simple sphere, cube or any
other Unity primitive can work just fine by just using an image attached to it
as its texture.

When
creating your Alexa skill…make sure not to make any spelling mistake…otherwise
you will hit you head thinking why Alexa is doing what you’re asking her to do…

When
testing our your application both Debug.log() and Print() will become your best
friends…nothing better than a printed value or message to realize what going
wrong.

When
moving an object, always make sure to record its original position and then add
the new value to that recorded position. Otherwise, something might provoke the
values to go wrong…by having the original values recorded, you avoid having to
recalculate the position but just call that variable and get things where they
belong.

When
using 3D Text you will notice that even if you put another object in front of
it…it will be always visible…which is not very likely…so we have two
options…either create a shader to occlude it…or the easiest one…make the
material that it’s in front of it transparent. That’s not perfect for all
situations but at least will work.

The
biggest problems when making Unity and Alexa speak, is that when you ask Alexa
to turn on the lights, she will respond “The lights are on” …but then if you
ask a second time her response should be “The lights are already on” …to make
this…we should need to use a Database or something to store state
information…and when closing the application, we would need to clean up the
states…while this might be doable…it’s a lot of work, and what happens if the
application crashes? Would we need to go and reset the states manually? Not
ideal…

That
leads me to the point of using the elevator…you can open the doors or sent it
to any of the floors…for the main part…that’s easy…each floor is a scene, so
you need to be on the first floor in order to make the elevator to floor two or
three…but…what if you’re outside the elevator? You are on floor one…ask for
floor three…and then you open the door…as your characters moves along with the
elevator floor…when you open the door everything will look bad…solution? Simply
using a cube without a mesh renderer, so it’s invisible…assign a collider with
“is trigger” enabled…and validate that the player is colliding with the cube in
order to make the elevator move…that way, even you ask for floor three and
Alexa confirms that the elevator is going up…nothing will happen…when you open
the door…we can assume that the elevator went down or up to your floor in order
for you to hop in…just an illusion…but it works…

Alexa
doesn’t have an option to delay the re-prompt, so when exploring the Virtual
House she will ask you “What else can I do for you?” and if we don’t respond
the skill will just die…so we will need to wake her up again…that’s kind of sad
due to the nature of the application…but nothing to be done unless Amazon
releases a way on making the re-prompt to wait longer…

As
the whole Alexa-Unity3D relays on Heroku…expect some downtimes or responses
from Alexa that are not actually replicated in the virtual world…might be an
internet connection glitch or just Heroku glitch…

As I mentioned first…the environment gets affected by the
weather…if it’s sunny…you will see a sunny clear sky…if it’s rainy you will see
a dark and gloomy sky…and this involves using a Skybox…although not your
regular Skybox…and what is a Skybox, anyway? Well…simply put…is a cube that
covers your whole environment and has different images to simulate the
environment…the problems is that the regular Skybox only allows you to assign
six sides…which is of course not likely…you need to use a twelve side Skybox…then
you can assign sunny image and also cloudy images…that way when checking the
weather you can modify the luminosity and that will also affect the Skybox as
it will use one or the other giving that nice effect on reflecting the outside
weather…