Apple ARKit is going to be largest AR platform in the world when it gets launched later this year. Hundreds of millions of iPhone and iPad devices will be compatible with ARKit, thanks to the powerful CPU and GPU that are already available on these devices. To enable highly accurate scene detection and object tracking, you need to run on-device neural network-driven algorithms. These algorithms cannot be made run entirely on CPU just like any other algorithm since they require more power and consume more battery. Since Apple exactly knows the hardware inside these devices, they have optimised the ARKit to run well on the iOS devices in a power efficient way.

Google on the other hand cannot do the same since their Android ecosystem is built on wide variety of CPU and GPU models from different vendors like Qualcomm, Samsung, MediaTek, etc, and their capabilities varies dramatically. Qualcomm, the most popular mobile processor maker is trying to solve this problem using the Neural Processing Engine (NPE) SDK. The Snapdragon NPE is a deep learning software framework designed for Snapdragon Mobile Platforms. It helps developers run one or more neural network models trained in Caffe/Caffe2 or TensorFlow on Snapdragon mobile platforms, whether that is the CPU, GPU or DSP. It supports both convolutional neural networks and custom layers.

The Snapdragon NPE is engineered to help developers save time and effort in optimizing performance of trained neural networks on devices with Snapdragon.

It does this by providing tools for model conversion and execution as well as APIs for targeting the core with the power and performance profile to match the desired user experience.

Facebook for example is planning to integrate the Snapdragon NPE into the camera of the Facebook app to accelerate Caffe2-powered AR features. As a result, they can achieve 5x better performance on the Adreno GPU, compared to a generic CPU implementation resulting in a more fluid, seamless and realistic application of AR features when capturing photos and live videos.

APIs for controlling loading, execution and scheduling on the runtimes

Desktop tools for model conversion

Performance benchmark for bottleneck identification

Sample code and tutorials

HTML Documentation

The Snapdragon NPE is compatible with Snapdragon 600 and 800 Series Mobile Platforms. According to my estimate, there will be at least 200 million active mobile devices running Snapdragon 600 and 800 series processors. Hopefully, developers will take advantage of this NPE platform to create great AR experiences on mobile devices. You can download the SDK here.

On a related note, Microsoft earlier this week announced that the next version of HoloLens will feature the second version of the HPU that will include an AI coprocessor to natively and flexibly implement Deep Neural Networks. Read more about it here.

Some links in the article may not be viewable as you are using an AdBlocker. Please add us to your whitelist to enable the website to function properly.