Advertising agency kbs+ asked us to join their team in creating a foot-stopping augmented reality experience that would bring the BMWi Concept Vehicle to the streets of New York City. By combining our expertise in computer vision and real-time graphic processing with the conceptual vision of kbs+, the Window Into the Near Future was born.

Displayed on a 6th avenue storefront along Bryant Park, this project transformed live traffic into BMWi Concept Vehicles… except the iconic New York taxi of course, those were left in place. On foot or in a car, people smiled with surprise at the future of the streets reflected back at them.

(video credit: kbs+)

Technology

BMWi: Window into the Near Future is built on two pieces of custom software (tracking and display), two computers, and three cameras. Each row of traffic needed to be tracked independently from a low angle camera. The tracking environment was one of the hardest imaginable. Constant changes in lighting conditions, fast moving shadows from nearby trees and buildings, hundreds of pedestrians that needed to be differentiated from cars, traffic of over a hundred cars per minute at times, and vastly different day and night environments.

Combined multi-threaded C++ and GLSL (GPU based Shader language for fast graphics processing) algorithms detected cars, while ignoring all other movement and changes in the environment. It then cataloged every car, determining whether it was a taxi or a normal car. This data was sent to a second computer/software which created the live display. This software constantly recorded the live feed from a camera which looked out the window. It determined what was foreground and what was background, then used eight layers of image processing per car to match environmental lighting, animate, and to add reflections to overlaid BMW i8 and i3 cars. All of this happening at an impressive 120fps, allowed for a perfect mirror of the street traffic.

cars_count

Engagement

The software generated real time metrics for all cars and pedestrians that passed by the display. The car metrics were then tied into the BMW i series campaign by being used to calculate potential savings in CO2 emissions, fuel consumption, and fuel costs. These savings were displayed live on the screen and could be seen increasing with each passing car. In addition, a content management system allowed for the display of pre-screened tweets.