How To Build A Virtual Reality Model With A Real-Time Cross-Device Preview — Smashing Magazine

by · November 8, 2018

About The Author

PhD in artificial intelligence at UC Berkeley, with a focus on small neural networks in perception, for autonomous vehicles. Big fan of cheesecake, corgis, and …
More about Alvin

In this tutorial, we’ll program three-dimensional objects and add simple interactions to these objects. Additionally, you can learn how to build a simple message passing system between clients and servers.

Virtual reality (VR) is an experience based in a computer-generated environment; a number of different VR products make headlines and its applications range far and wide: for the winter Olympics, the US team utilized virtual reality for athletic training; surgeons are experimenting with virtual reality for medical training; and most commonly, virtual reality is being applied to games.

We will focus on the last category of applications and will specifically focus on point-and-click adventure games. Such games are a casual class of games; the goal is to point and click on objects in the scene, to finish a puzzle. In this tutorial, we will build a simple version of such a game but in virtual reality. This serves as an introduction to programming in three dimensions and is a self-contained getting-started guide to deploying a virtual reality model on the web. You will be building with webVR, a framework that gives a dual advantage — users can play your game in VR and users without a VR headset can still play your game on a phone or desktop.

Developing For Virtual Reality

Any developer can create content for VR nowadays. To get a better understanding of VR development, working a demo project can help. Read article →

In the second half of these tutorial, you will then build a “mirror” for your desktop. This means that all movements the player makes on a mobile device will be mirrored in a desktop preview. This allows you see what the player sees, allowing you to provide guidance, record the game, or simply keep guests entertained.

Prerequisites

To get started, you will need the following. For the second half of this tutorial, you will need a Mac OSX. Whereas the code can apply to any platform, the dependency installation instructions below are for Mac.

Internet access, specifically to glitch.com;

A virtual reality headset (optional, recommended). I use Google Cardboard, which is offered at $15 a piece.

Step 1: Setting Up A Virtual Reality (VR) Model

In this step, we will set up a website with a single static HTML page. This allows us to code from your desktop and automatically deploy to the web. The deployed website can then be loaded on your mobile phone and placed inside a VR headset. Alternatively, the deployed website can be loaded by a standalone VR headset. Get started by navigating to glitch.com. Then,

Click on “New Project” in the top-right.

Click on “hello-express” in the drop-down.

(Large preview)

Next, click on views/index.html in the left sidebar. We will refer to this as your “editor”.

(Large preview)

To preview the webpage, click on “Preview” in the top left. We will refer to this as your preview. Note that any changes in your editor will be automatically reflected in this preview, barring bugs or unsupported browsers.

(Large preview)

Back in your editor, replace the current HTML with the following boilerplate for a VR model.

To preview this on your VR headset, use the URL in the omnibar. In the picture above, the URL is https://point-and-click-vr-game.glitch.me/. Your working environment is now set up; feel free to share this URL with family and friends. In the next step, you will create a virtual reality model.

Step 2: Build A Tree Model

You will now create a tree, using primitives from aframe.io. These are standard objects that Aframe has pre-programmed for ease of use. Specifically, Aframe refers to objects as entities. There are three concepts, related to all entities, to organize our discussion around:

Geometry and material,

Transformation Axes,

Relative Transformations.

First, geometry and material are two building blocks of all three-dimensional objects in code. The geometry defines the “shape” — a cube, a sphere, a pyramid, and so on. The material defines static properties of the shape, such as color, reflectiveness, roughness.

Aframe simplifies this concept for us by defining primitives, such as <a-box>, <a-sphere>, <a-cylinder> and many others to make a specification of a geometry and its material simpler. Start by defining a green sphere. On line 19 in your code, right after <!-- start code here -->, add the following.

Second, there are three axes to transform our object along. The x axis runs horizontally, where x values increase as we move right. The y axis runs vertically, where y values increase as we move up. The z axis runs out of your screen, where z values increase as we move towards you. We can translate, rotate, or scale entities along these three axes.

For example, to translate an object “right,” we increase its x value. To spin an object like a top, we rotate it along the y-axis. Modify line 19 to move the sphere “up” — this means you need to increase the sphere’s y value. Note that all transformations are specified as <x> <y> <z>, meaning to increase its y value, you need to increase the second value. By default, all objects are located at position 0, 0, 0. Add the position specification below.

Third, all transformations are relative to its parent. To add a trunk to your tree, add a cylinder inside of the sphere above. This ensures that the position of your trunk is relative to the sphere’s position. In essence, this keeps your tree together as one unit. Add the <a-cylinder> entity between the <a-sphere ...> and </a-sphere> tags.

Reload the website preview on your VR headset, and check out your new tree. In the next section, we will make this tree interactive.

Step 3: Add Click Interaction To Model

To make an entity interactive, you will need to:

Add an animation,

Have this animation trigger on click.

Since the end user is using a virtual reality headset, clicking is equivalent to staring: in other words, stare at an object to “click” on it. To effect these changes, you will start with the cursor. Redefine the camera, by replacing line 13 with the following.

The above adds a cursor that can trigger the clicking action. Note the objects: .clickable property. This means that all objects with the class “clickable” will trigger the animation and receive a “click” command where appropriate. You will also add an animation to the click cursor, so that users know when the cursor triggers a click. Here, the cursor will shrink slowly when pointing at a clickable object, snapping after a second to denote an object has been clicked. Replace the comment <!-- add animation here --> with the following code:

The above properties specify a number of configurations for the animation. The animation:

Is triggered by the click event

Modifies the tree’s position

Starts from the original position 2 0.75 0

Ends in 2.2 0.75 0 (moving 0.2 units to the right)

Animates when traveling to and from the destination

Alternates animation between traveling to and from the destination

Repeats this animation once. This means the object animates twice in total — once to the destination and once back to the original position.

Finally, navigate to your preview, and drag from the cursor to your tree. Once the black circle rests on the tree, the tree will move to the right and back.

Large preview

This concludes the basics needed to build a point-and-click adventure game, in virtual reality. To view and play a more complete version of this game, see the following short scene. The mission is to open the gate and hide the tree behind the gate, by clicking on various objects in the scene.

(Large preview)

Next, we set up a simple nodeJS server to serve our static demo.

Step 4: Setup NodeJS Server

In this step, we will set up a basic, functional nodeJS server that serves your existing VR model. In the left sidebar of your editor, select package.json.

With the utilities loaded, the provided server next instructs the server to return index.html as the homepage. Note there is no new code written below; this is simply an explanation of the existing source code.

Once you are finished editing, Glitch automatically reloads the server. Click on “Show” in the top-left to preview your application.

Your web application is now up and running. Next, we will send messages from the client to the server.

Step 5: Send Information From Client To Server

In this step, we will use the client to initialize a connection with the server. The client will additionally inform the server if it is a phone or a desktop. To start, import the soon-to-exist Javascript file in your views/index.html.

Next, we will define a series of initial messages to exchange with the server side. Define a new socket.io object to represent the client’s connection to the server. Once the socket connects, log a message to the console.

This is the first of simple message-passing, where our client sends information back to the server. Quit the running NodeJS process. For the final part of this step, we will have the client send camera information back to the server. Open public/client.js.

Once again, load the preview on your desktop and on your mobile device. Once a mobile client is connected, the server will immediately begin logging camera position and rotation information, sent from the client to the server. Next, you will implement the reverse, where you send information from the server back to the client.

Step 6: Send Information From Server To Client

In this step, you will send a host’s camera information to all mirrors. Open your main server file, server.js.

The broadcast modifier ensures that the server sends this information to all clients connected to the socket, except for the original sender. Once this information is sent to a client, you then need to set the mirror’s camera accordingly. Open the client script, public/client.js.

Here, check if the client is a desktop. If so, receive the move data and log accordingly.

Load the preview on your desktop and on your mobile device. In your desktop browser, open the developer console. Then, load the app on your mobile phone. As soon as the mobile phone loads the app, the developer console on your desktop should light up with camera position and rotation.

Open the client script once more, at public/client.js. We finally adjust the client camera depending on the information sent.

Load the app on your desktop and your phone. Every movement of your phone is reflected in the corresponding mirror on your desktop! This concludes the mirror portion of your application. As a desktop user, you can now preview what your mobile user sees. The concepts introduced in this section will be crucial for further development of this game, as we transform a single-player to a multiplayer game.

Conclusion

In this tutorial, we programmed three-dimensional objects and added simple interactions to these objects. Additionally, you built a simple message passing system between clients and servers, to effect a desktop preview of what your mobile users see.

These concepts extend beyond even webVR, as the notion of a geometry and material extend to SceneKit on iOS (which is related to ARKit), Three.js (the backbone for Aframe), and other three-dimensional libraries. These simple building blocks put together allow us ample flexibility in creating a fully-fledged point-and-click adventure game. More importantly, they allow us to create any game with a click-based interface.

Here are several resources and examples to further explore:

MirrorVRA fully-fledged implementation of the live preview built above. With just a single Javascript link, add a live preview of any virtual reality model on mobile to a desktop.