Embedded Vision: Who's Watching Whom & Why

Imaging technology is no longer just about the never-ending megapixel race among CMOS image sensors. As market focus shifts to "vision" processing, the industry has drawn a new battle line.

TOKYO — Imaging technology is no longer just about the never-ending megapixel race among CMOS image sensors. As market focus shifts to "vision" processing, the industry has drawn a new battle line -- over how fast and how accurately a processor can capture, dissect, and interpret data in a manner comprehensible to an embedded system.

In short, the whole concept of who's watching whom has flipped.

In the embedded vision world, what matters is not so much you, the photographer, who wants to take better photos; instead, the technology now exists to cater to embedded systems that need to watch you, recognize who you are, analyze your behavior, and process data they think you need.

You might call this just the plain reality of technology progress in machine vision or computer vision. Maybe so. But I confess that some of the embedded vision plots hatched by marketers today are disturbing enough to make me cringe.

None of this stuff, of course, is more worrisome than the NSA's electronic spying programs. But the very notion of a bunch of sensors physically watching me -- solely to make a commercial gain at my expense -- gives me, at least, a slight case of the willies. At worst, it's a reminder of the increasingly Orwellian society we already live in.

Over a cup of coffee in Tokyo, I recently sat down with Tom Wilson, vice president of business development at CogniVue, a Quebec-based embedded vision technology developer. Wilson tried to convince me that automotive isn't the only market being targeted by vision processing technology developers like CogniVue.

Here are a few examples he shared with me -- in terms of what comes next with embedded vision:

Drive a car on a deserted road in the dark. Street lamps -- normally switched off -- light up the road just in front of your car, as you move forward. As soon as they sense your car is leaving, they go off. (Yeah, I know: an evening's drive through The Twilight Zone.)

Walk in front of a digital sign -- a gigantic electronic display in a public space. The sign, even before you notice it, recognizes your gender and age, then quickly changes the ad message -- to fit your demographic profile -- as you look at it. (Yeah, I know: shades of Minority Report.)

Smartphones that can recognize your hand gestures, or that can do face recognitions to help you tag images (by informing you who you are seeing, and whose pictures you are taking, and even uploading to social networks.)

A set-top box embedded with eyes in your living room identifies who is watching what program. It sends the information to a backend server, triggering a digital product placement in a TV program. (Right. Saw that in Fahrenheit 451.)

Among these examples, what ticked me off was the last item about a set-top box with eyes. Of course, for someone who's known Kinect (a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs), I probably shouldn't have been so surprised. But I needed further clarification over what it exactly does.

"Say you are watching Friends. The set-top box knows you're watching it and you actually like Pepsi instead of Coke," explained CogniVue’s Wilson. The backend server, then, can digitally insert a Pepsi can, replacing a Coke, in Monica's living room.

Wilson pointed out that Mirriad, a developer of ad platforms, is one company working on such a project. "The plan is to couple this type of ad insertion with viewer preference," he explained. In fact, a set-top box with eyes isn't such a far-fetched idea. Mirriad recently signed a deal with Pace, a set-top box vendor, to trial this in the UK, according to Wilson.

While explaining the digital product placement scheme, Wilson joked that this is partly why he doesn't own a TV. But he made sure that I understood the far-reaching ramifications of embedded vision applications and how the competition among embedded vision IP vendors -- both software and hardware -- has been escalating in recent years.

If I remember it correctly, a similar system was used a couple of decades ago for deciding the TV program viewership ratings. A camera built into the STB would record the information about what programs are watched at what time. The surveyors would then collect the stbs from homes, physically and collate the viewership information.

Now that we have good communications back bone and the image processing capability , many of such innovations are possible.

The question still remains about the privacy of the viewer , because such things are done without the knowledge of the person standing in front of the camera.

But as Rich@ was saying, I would not like any box in my living room watching me, period. Especially, when they plan to use whatever information about me for their commercial gain.

I think we are so jaded about our own privacy these days -- we tend to let technology designers and marketers do whatever they want to do about data they collect about us. I want boundries, and I want to let technology developers and marketers know what's NOT acceptable to consumers.

As to the system watching me at home, though, it had better not be doing that without my express permission. I would not welcome such intrusion as the kind of inserted brand awareness stuff based on my personal characteristics that you described.

"Digital product placement" in a TV program might feel like such a trivial thing to some people, but to me, it's definitely an intrusion. I am surprised a content owner would not object to that.

I am fully aware of the vending machine you described -- a huge LCD kind. I have been also told that the machine is embedded with an image sensor. I haven't used the damn machine enough times to notice the different selections it offers, though.

Quoting from the article: "Walk in front of a digital sign -- a gigantic electronic display in a public space. The sign, even before you notice it, recognizes your gender and age, then quickly changes the ad message -- to fit your demographic profile -- as you look at it. (Yeah, I know: shades of Minority Report.)"

Junko, you are currently in Tokyo? Go to any of the major JR train stations and look for one of the new drink vending machines (the ones with a huge LCD display). They do exactly what you describe. Based on image recognition the machine analyzes the potential customer standing in front of it and comes up with a suggested drink which might suit your taste. I was once told that the machine is even self-learning, adapting its proposals based on what previous customers actually chose.

Junko, it seems like the kind of thing you are talking about is the use of security cameras in stores to also provide information about people's movements that help store planners see what displays are attracting attention. For instance, the system can determine rough age and gender of the people passing by, then measure the time they linger in front of displays as a means of measuring market interest and the like. Right?

As stores are public places, and the same thing can happen if store employees were keeping their eyes open, I dont see that as being too creepy.

As to the system watching me at home, though, it had better not be doing that without my express permission. I would not welcome such intrusion as the kind of inserted brand awareness stuff based on my personal characteristics that you described. I might even change brands to protest any company doing that.

I guess there's much unexpected video-ing going on these days that this didn't strike me as particularly worrisome. I mean, parking lots and elevators, for instance. You have to expect that you're being monitored and recorded always. Also, tablets and laptops may use face recognition as a security measure, to prevent theft.

So what the hey, Junko. If the set-top box uses face recognition for something as trivial as ad placement, I guess that doesn't sound like an unexpected innovation? Although I'd certainly prefer if the add placement occurs only inside your home (e.g. multiple ads downloaded, selection made only at the STB), without having to send the images to some outside server.

You may call this is no different from cookies on websites, which keeps tracking what you are reading on the web.

That's true.

But embedded vision has so much potential to offer security and safety features (surveillance cameras, ADAS, etc.). I think it's a shame if the"vision" industry entirely ignores the privacy issues. It's time for the stakeholders to start talking about the best practices or basic guidelines for embedded vision.

Wait. Why "freak you out"? How is this different from, say, security cameras that are just about ubiquitous these days? Or are you saying that the images they show from inside your living room or kitchen, would be made available to anyone outside?

I expected the video to show something about vision algorithms, but unless I watched the wrong video, that's not what the video shows. It just shows candid video of people inside and outside.