Rob MarvinHow to Build Augmented Reality Apps for the Industrial IoTEnterprise software company PTC explains how to use its Vuforia and Thingworx platforms to create 3D augmented reality (AR) models for industrial Internet of Things (IoT) devices.

Enterprise augmented reality (AR) has applications across a broad array of businesses and industries. Marketers and salespeople can use AR as a 3D e-commerce tool to model products for customers in real space. Surgeons can wear AR glasses to get patient data and overlaid diagrams on a heads-up display, keeping their hands free during procedures. But for enterprises with heavy manufacturing operations, the most game-changing AR use case is in the industrial Internet of Things (IoT).

Field workers and technicians on factory floors can leverage 3D AR modeling combined with real-time data from IoT sensors in a number of ways. By using an AR app on smartphones, tablets, or enterprise AR headsets, workers can monitor connected devices for issues and perform routine maintenance far more easily than inspecting machines that can sometimes be difficult or dangerous to reach. AR headsets such as Google Glass Enterprise and Microsoft HoloLens can also display immersive, step-by-step online training and tutorials for assembling and operating machinery that would otherwise require reading thick technical manuals and instructions. For engineers, plant managers, or anyone working with complex machinery—be it an automobile assembly line, oil and gas piping, or a jet engine—enterprise AR is redefining how industrial workers do their jobs.

The tough part for businesses is crafting an AR experience to fit the unique needs of their employees and the industrial IoT devices with which they're working every day. Enterprise software company PTC has a wealth of experience on this front. PTC owns both Vuforia, the popular AR development platform previously owned by Qualcomm, and leading IoT development platform ThingWorx.

PTC has spent the past several years building out and integrating the two platforms into a number of tools that make up an end-to-end AR IoT pipeline. The solution is completely customizable depending on a company's hardware, software, and cloud infrastructure needs. PCMag met with Mike Campbell, Executive Vice President of the ThingWorx platform at PTC, to break down how ThingWorx and Vuforia work together. Campbell explained how businesses can incorporate all of their existing data and devices into AR apps. He also demonstrated how the company's recently launched ThingWorx Studio melds Vuforia's AR development capabilities with 3D modeling tools for the industrial IoT.

1
Bringing AR to IoT

In the recently launched ThingWorx 8, the latest version of PTC's industrial IoT platform, the company brought more of Vuforia's AR app development capabilities directly into ThingWorx. Through a free trial program of ThingWorx Studio, PTC has given more than 2,000 participating companies access to a native AR authoring and publishing environment within ThingWorx. Companies piloting the capability include car makers such as Tesla, Toyota, and Volkswagen, heavy machinery producers including Caterpillar and John Deere, tech and consulting giants, and even NASA and the US Army.

"This is a capability that anyone will be able to use," said PTC's Campbell. "We put this right in Thingworx Studio. So, if you make AR experiences for tripods, you can create a target for the tripod that will look like a wireframe view and then author anything you want against that. Drag and drop and, boom, you’ve got AR."

2
Data Flexibility

Thingworx Studio is purpose-built for AR but lets enterprises reuse existing 3D assets, making it easy to connect all of their factory's existing machinery into the IoT ecosystem. The platform can also handle existing cloud data sources, and packs native cloud IoT integrations with Amazon Web Services (AWS), Microsoft Azure, and GE Predix.

"Every machine and part in a factory was modeled in CAD before it was manufactured. We can take that CAD and re-purpose catalogs of customer components. Any 3D data, no matter where it comes from, we can reuse," said Campbell.

"We can also make it easier to get data from all the controllers, sensors, and gateways. Every machine has its own protocols and languages, so we've acquired and integrated technology allowing us to talk to all of those machines. We support more than 150 different protocols used by thousands of machines, so we can walk into a factory and begin getting data onto Thingworx in less than an hour. We've got a device that looks like a small metal hockey puck that you can stick on a machine, and it starts transmitting data like vibrations, temperature, etc., right into ThingWorx."

3
Step-by-Step Instructions

Guided AR instructions for assembling and dissembling, inspecting, and maintaining intricate systems is a primary app for the industrial IoT. It's also a task that can become repeatable and standardized. In ThingWorx 8, PTC also released three pre-built apps—ThingWorx Controls Advisor, ThingWorx Asset Advisor, and ThingWorx Production Advisor—for typical industrial tasks such as remote asset monitoring and managing real-time production performance.

Campbell explained how to create repeatable, step-by-step instructions while respecting the unique mechanics of any given machine. He also said ThingWorx is working on rolling out CAD-based tracking in the next few months to eliminate the need for placing physical markers in front of an AR lens to anchor the model.

"We can reuse step-by-step instructions incorporating IoT data and make it available via the cloud, with tools that work a little bit like PowerPoint animations," said Campbell. "Grab something, drag it to where you want, and take a snapshot. Then, when creating the instructions, you can specify things like 'unscrew this' with a prepackaged animation that rotates the screw counter-clockwise and slides it out."

"There's intelligence built in there, so it's not just drag and drop," Campbell continued. "There's also respect for kinematics. So, when you design something, you specify that this is a certain kind of joint or slider and it will respect that as well. In a training environment, we're taking these same concepts—a lot of manufacturing and inspection—and putting things together in a way where you don't have to look at a schematic because you see an overlay in front of you saying 'these three wires need to be plugged in here, here, and here.'"

4
Native Anomaly Detection

Native anomaly detection means real-time analytics on an IoT device that can help an engineer or inspector perform predictive maintenance. The ThingWorx Asset Advisor app, for instance, remotely monitors physical assets in real time to automatically detect anomalies and trigger alerts to improve efficiency and quality.

"So you’ve connected your machine on the factory floor," said Campbell. "What you can do is tell Thingworx 'learn what normal is' so then, when something is not normal, it triggers some event, popping up an alert in the AR app or directly in one of your integrated business systems."

5
Create Digital Twins

The concept of native anomaly detection also factors into the idea of a "digital twin" for machines on your factory floor or out in production. One of the most powerful capabilities of bringing AR into industrial IoT is the ability to create digital representations of your physical machines. Campbell explained how to create a digital twin in ThingWorx Studio that pulls in real-time IoT data for that machine, demonstrating with a 3D model of a motorcycle.

"I've got a physical machine in my factory," said Campbell. "What ThingWorx allows you to do is create a digital equivalent of that, which you can then synthesize, analyze, and get insight to drive action, either by orchestrating business systems like kicking off an action in a CRM system, or trigger something that affects the product itself."

"In ThingWorx Studio, we have resources over here, a palette in the middle, and properties on the right. I can place the 3D model of the bike and then author our AR experience against it," he demonstrated. "I'm placing virtual gauges, applying the step-by-step instructions, and making that available with a button over here that I can switch between a 3D view and a 2D environment that you'd see through an iPad screen. Then, instead of showing a gauge with no numbers, we're actually bonding the value shown to data coming out of ThingWorx to create this digital twin where you're seeing the battery data, oil levels, tire pressure, temperature—all in real time."

6
Mixed Reality Content Creation

For developers and AR content creators, the value in Vuforia is the ability to create experiences for any form factor. Available through ThingWorx Studio, the platform now supports AR IoT apps for Android, iOS, Windows 10, Microsoft HoloLens, and the broader Windows Mixed Reality ecosystem, and other digital eyewear devices such as the enterprise AR glasses offered by ODG and Vuzix.

Campbell discussed Microsoft HoloLens, in particular, as a powerful medium for the kind of enterprise content and IoT apps created with ThingWorx and Vuforia. When I strapped on a HoloLens to take a look for myself (pictured a few slides earlier), I saw the same AR motorcycle model in front of me, with HoloLens-specific gesture controls and visual instructions. Campbell also loaded a demo that let me put together and take apart all of the components in a generator.

"We want to unleash the industrial enterprise's 3D content on HoloLens and all these devices," said Campbell. "Look at the augmented work instructions to replace the air filter in a Caterpillar generator. These kinds of interactive training and service use cases range from marketing and selling (what this generator looks like and how it fits in my work site) to how to operate it, service it, and build that generator for the first time. You can take it apart and put it back together with an air tap."

"The folks at Microsoft are excited about what we’re doing because they have a problem. They’ve got these awesome new devices, but the challenge remains getting content created for them," Campbell went on. "That’s why they introduced 3D Paint and all of these things. From a PTC perspective, that’s just hardware. We’re all about providing the software and a platform to power that software, so we don’t really care what you want to consume it on; iPhone, iPad, Android device, it doesn’t matter. We’re all about making it stupid-simple to get that content out there."

7
Putting It All Together

One of the biggest challenges with industrial IoT is standardizing data on an enterprise scale. Large corporations and manufacturing operations need to hook these IoT endpoints into all of their existing systems and set up a data pipeline to take action on all of that data. Otherwise, the AR experience isn't tied to tangible value and can't bring ROI to the business.

For a platform such as ThingWorx, that means melding hardware, software, and data integrations into a holistic solution that every company can customize to its needs. For industrial IoT and AR to work together, you need tools that will take all of the manual complexity out of it for businesses.

"If you have nothing, we’ve got a top-to-bottom stack," said Campbell. "If you’ve got machines that have already been sensored, a device with telematics, or you’ve already got product data in the cloud with Azure or AWS, we can help you do things with that to drive business insight and specific use cases."

"What we’re finding is that AR is a new way to communicate, a new language, a new medium of communication," Campbell continued. "We know how to build desktop apps, web apps, and mobile apps, but no one knows how to build AR apps. It’s a whole new world. You’ve got to be able to iterate quickly. And spending thousands of dollars and man-hours only to get something not quite right is pretty painful. The ThingWorx Studio tool allows you to quickly crank that stuff through, test them out, and optimize them."

Get Our Best Stories!

This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.