The technology behind Lady Gaga's Grammy tribute to David Bowie

Viewers who tuned in to watch last night's Grammy Awards expected there would be a tribute to the iconic singer David Bowie, who passed away in January. But what they got was so much more. Lady Gaga performed a medley of Bowie's greatest hits in a dazzling high-tech video tribute that has gone viral this morning.

Lady Gaga teamed up with Intel and used the company's RealSense 3D cameras to come up with the effects. The technology was developed over the course of several years by teams at Intel's California and Israel R&D labs. Among other things, the performance featured a hologram of Bowie himself and an accompaniment from a piano with three robot arms. You can see the result of their collaboration in the video below:

https://www.youtube.com/watch?v=3fGBZhsa4VU

Intel's RealSense technology basically allows cameras to see in 3D. This helped the images being displayed mold to the contour's of Lady Gaga's face. "We introduced ‘living canvas’ technology to her, which allows her to basically have what we call digital skin – which has been used in tech art installations, but never before for a live performance," Paul Tapp, Intel’s director of technology, explained to Vanity Fair.

"Once we’ve got the tracking in place, those cameras and markers are telling the computer exactly what angle and rotation and axis her face is and her facial expressions. Are her eyes opened or closed? Is her mouth happy or sad? Are her cheeks up or down? We have to track all of that with very intricate detail and then in real time we have to adjust that digital makeup that we’ve computer generated, get it sorted according to how her facial position is, and project it exactly to the right place at the right time.”

Work on the RealSense cameras began back in 2011 when Intel started working with Omek Interactive. The Israeli startup, which wrote software for 3D cameras, was founded by Brown University graduate Gershom Kutliroff and later acquired by Intel.

Once inside the Intel family, work on the camera further developed to include facial analysis, hand and finger tracking, speech recognition, and augmented reality. The project was spearheaded at Intel Israel's headquarters by Mooly Eden, an alumnus of the Technion Institute of Technology in Israel. "We’ve taken something from science fiction and removed the fiction,” said Eden.

https://youtu.be/DuUVIat40YU

As for Lady Gaga, she sees the Grammy performance as the perfect tribute to Bowie. "It is important for me to continue to build upon something that he was truly the first person to do. The first person to bridge music, fashion, and technology," she told Mashable. "These things were at a cultural crossing in Bowie."

Gaga took her adoration for the late singer one step further. Shortly before her Grammy performance, she got a tattoo of Bowie. "This was the image that changed my life," she wrote on Snapchat.