12 Views of Facebook’s Dev Con

SAN JOSE, Calif. — Facebook is pouring a new software foundation for machine learning and hammering on top of it the early 2x4s of augmented and virtual reality. Its annual developer conference here showed a rich mix of creative and sometimes crude work toward its many ambitious aims.

Under the hood, Facebook and its cloud computing rivals are essentially global giant data farms where harvesting the good stuff depends on machine learning. Facebook’s engine is unique in part for how it aims to leverage both its global warehouses of servers, and its users’ smartphones.

Facebook’s new Caffe 2 framework takes a page -- and a key developer -- from rival Google’s Tensorflow. Caffe 2 is a significantly upgraded version of the machine-learning framework originally created at UC Berkeley by Yangqing Jia. After graduating, he spent two years at Google working on Tensorflow and other projects before Facebook hired him in February 2016 as engineering lead for its AI platform.

“We needed more flexibility, so we made [Caffe] more modular and friendly to different hardware back ends…mobile is a major interest,” said Jia in an interview with EE Times on the F8 show floor.

Smartphones will use Caffe 2 models to recognize and enhance objects in photos, creating AR effects similar to Pokemon Go and Snapchat filters. The so-called “style transfers” mark Facebook’s first strategic steps to draw consumers into AR.

On the show floor, Qualcomm showed neural-network software handling image recognition at 50 frames/second on the Adreno GPU cores in its high-end Snapdragon SoCs. That’s much faster than the 12 f/s the SoC’s Kryo CPU delivers as a default. This summer Qualcomm will release the software in a version that also uses its Hexagon DSP as an accelerator.

Facebook is clearly betting on such capabilities being widely available in the future. For today it will have to deal with a widely fragmented handset market that lacks such support.

Processor vendors will want to support all broadly used machine learning frameworks, but their efforts will take time. For example, ARM and Ceva, whose cores are widely used in smartphones, have so far expressed support for Tensorflow, not Caffe 2. Intel, Nvidia and Qualcomm were quick to say they will support Caffe 2. Facebook already uses Nvidia’s GPUs on it training servers.

“We see [potential for] a lot of hardware optimizations…There are quite a few computation patterns stabilized enough for hardware to use, and more and more computation patents will stabilize,” he said.

However, all the frameworks are still evolving to support a wider range of neural networks with higher performance. “It’s a healthy competition, in general we’re all searching for a better solution for A.I.—it’s like the evolution of programming languages,” he said.

For its part, Facebook is replacing a mix of frameworks including Torch with Caffe 2. Facebook made the framework open source and will use it on all its machine learning jobs including computer vision and machine translation on servers. It also is working with rivals Amazon and Microsoft to make it easier for business users to tap into their AI services.

“It’s about setting up development environments easily, similar to using pre-installed software,” Jia said.

For its part, Google’s Tensorflow is being used by AirBnB, Dropbox, SAP, Twitter, Uber and Xiaomi.

Overall, Facebook is behind its Web rivals in machine learning, according to Richard Windsor, analyst at Edison Investment Research. “AI remains essential to Facebook’s long-term growth as it is sitting upon a mountain of data but still is not in a position to really make the most of the insights and automation that it can provide,” Windsor wrote in a research note that cited progress in image recognition where the company has made significant hires.

“Facebook had to move beyond Torch, which they use internally for research, since Torch is based on the Lua language,” said Karl Freund, a senior analyst at Moor Insights and Strategy. “Most AI programmers use and prefer Python, which is native for many other frameworks, including Caffe,” he said.

“The largest cloud and Internet sites have developed and promoted their own AI frameworks to keep a competitive edge and develop a loyal ecosystem,” Freund said. “They cannot use someone else’s open source versions as these always lag the code used internally by as much as a year. Each company will have their specific area of focus, for example, Facebook focuses on imaging and Amazon focuses on natural language processing,” he added.

Indeed, Facebook AI specialist Joaquin Candela showed how its Mask R-CNN algorithms can now tightly detect and classify people and objects—even when they are moving in video or have parts blocked by other objects.

Candela claimed the enhancements in Caffe 2 provided 100x speedups in inference jobs run on smartphones. Its use in Facebook’s image and texting apps — Instagram and Messenger — “makes it the largest A.I. deployment ever,” he said in a keynote.