Posted
by
CowboyNeal
on Saturday June 10, 2006 @09:33AM
from the tuning-in dept.

segphault writes "Ars Technica reports on a paper (PDF) about ambient audio analysis authored by Google researchers. The system described in the paper can effectively determine what television show a user is watching just by capturing a short audio clip. The paper explains how a regular computer microphone can be used to record an audio clip that is then converted into a statistical data summary and transmitted to a remote server which matches the clip against archived data in order to ascertain which TV show it is associated with. Apparently, the system is fully viable, and other kinds of ambient noise don't negatively impact its accuracy. The paper also describes how web services can provide contextually relevant information based on a consumer's television viewing activities."

Or maybe it can be used like those trucks in V for Vendetta that just went around the streets with directional mics recording small snippets of audio to see what the people (in general) were talking about on the streets. A way to check what channels are being seen the most in different areas of the cities?. Not that I'd like to have it on my neighbourhood any better than on my room, though.

Well, it's not Big Brother but Big Brother could get a court order to hear what was going on in your house at anytime since the microphone was turned on.
There is no way I'm letting Google or anyone else have an open mic. in my home.
Even if it were just turned on for brief periods at random intervals.
Why would anyone allow this in their livingroom? No, I didn't RTFM.
From the Paper (and mentioned in TFA if you read it):
The viewer's acoustic privacy is maintained by the
irreversibility of the mapping

That is a better system than what I first imagined. Still the encoding is being done in software so it would not be difficult to switch from encoded to raw without anyone knowing (Only with the request^W court order from an official agency of course).

From The Fine Paper;...the system can easily be designed to use an explicit 'mute/un-mute' button, to give the viewer full control of when acoustic statistics are collected for transmission.

There's a system in the UK where you can go out clubbing, here a song you like, dial a number and hold the phone out to the music and it'll text you the name of the song.
Assuming they don't hire scores of extremely knowledgable music buffs with quick fingers, surely it's a very similar system. TV dialogue may be less distinctive to the human ear but to a computer it just means a larger amount of data to search through.

Not quite, the service you mentionned recognizes a sound clip against its more or less exact replica in the (large) database.This here matches a sound clip to a pattern to find the TV show, meaning it doesn't have all the current episodes of the program in its database, it just has statistical data and patterns which help it match the audio. The latter could successfully match new (live) episodes without having the database updated. Your tune system wouldn't.

In order to perform its function, the audio-database
server must have access to a database of broadcast
audio data. However, the actual audio stream does
not need to be stored. Instead, only the compressed
representation (32-bit descriptor) is stored. This
allows as much as a year of broadcast fingerprints
to be stored in less than 1 GB of memory.

It sounds like they are effectively building a database to compare the recordings to. (even if these are not the actual material but

I always wanted to have the ability to "hash" songs, and come up with an algorithm that would be robust enough to work across multiple codecs and encoding options, different (relative) normalizations, and maybe even be able to handle empty space at the beginning and/or end of the song. (I would think by making a hash based on values relative to sound signatures within the clip this might be possible, but I don't really know how this stuff works.)

I always wanted to have the ability to "hash" songs, and come up with an algorithm that would be robust enough to work across multiple codecs and encoding options, different (relative) normalizations, and maybe even be able to handle empty space at the beginning and/or end of the song.

It's been done. Here's a system where you can hum a tune and it tells you the song: http://www.musipedia.org/ [musipedia.org]

Current systems are mostly based on pitch changes, so they aren't perfect (especially with the recycled slush turned out by low-grade high-visibility pop acts), and largely useless for rap, but they mostly work. There are numerous variations on the system, this is just one of the more significant ones that is publically availabel on the web.

I would think by making a hash based on values relative to sound signatures within the clip this might be possible, but I don't really know how this stuff works

What google is doing may or may not be related. They might instead be using a form of speech recognition technology, or a combination of both, or something else entirely.

Two acquaintances of mine work at the company you mentioned. What is also amazing is that they record the Cd's to disc creating a finger print of the Cd's. They then sell the Cd's off at cost price. They are really cheap, as they by in bulk and even get tons of freebies. They are legally allowed to do this as keeping data of a fingerprint of the CD does not constitute keeping the actual CD. What is also amazing is that the fingerprint allows for the song to be playing at a slower or faster rate and it will

I can't really comment on american shows but people get the same impression here in the UK, however the audio level actually stays the same, the audio is compressed to enhance the volume of the commercials. It has now become common practice to boost the sound level of quieter passages so there is more sound power in the frequency range where the ear is sensitive. Because most TV programs do not use audio compression in the same manner, the result is that commercials are often perceived as louder. In actual

if you want to check what tv show it is - it means that you like it, and want to watch later/tell your friends about it. Google can sell data about tv shows popularity to interested parties, that will know where to plant ads (or they could place ads themselves...). It also can be used to determine an ad price for a given show.

I'm not watching tv for 4 years now, it feels great. If I accidentally see some of it somewhere I'm shocked at how dumb it is.

I think this is exactly right.If Google really wants to "do no evil", then they need to use this technology to recognize that a commercial just started, and turn the darn volume down! I'm a reasonable guy - they don't need to turn the volume off or skip over the commercial (although they are welcome to do this), just turn it down to the point where the overcompressed signal is not blasting me out of my brain. It's almost impossible to quietly watch any 10PM network TV show without getting blasted by the c

Do you remember the MTV show Popup Video? They showed older music videos with popup balloons that gave extra information, like actors in the video that later became famous or mistakes made during production. If Google analyzed the sounds coming into your laptop and gave you a link to a site like the Internet Movie Database [imdb.com] then you could have Popup Television. Learn more about the specific episode you are watching, and even have the ability to edit that information yourself.

Is THIS why Google has been returning so many porn sites on my searches lately?

This exposes an example of how even sanitized information is not sanitized.
How can you return information on what people are watching and not return at least some very personal information.
Of course, some people might say "well, I don't watch that kind of stuff, so I don't mind", but the issue is subtle.
First, that attitude leads quickly to an assumption that there are two classes of people: people who don't mind being
tr

This seems like a not too complicated idea. You create an inexpensive operation that extracts what features you want from the sound data. Most importantly, you avoid features that are prone to randomness and entropy. It would take some research to figure out what the best features are and that's the audio fingerprint.

Since Google has more storage than you can imagine, they can most likely apply this fingerprinting technique to every episodes of every major show. Then they host the fingerprints in Google style and use their patented "Google Technology" to search it much the same way web content is searched.

Why would you want this? Well, there's the obvious marketing ploys. You know that people who watch Darma & Greg like to shop at Trader Joe's and like Odwalla brand food so you offer free episodes of Darma & Greg with only Trader Joe's & Odwalla episodes. You let the sponsors (Trader Joe's and Odwalla) foot the bill for the bandwidth/royalties or whatever.

The second useful implication would be cross suggesting shows to a user based on random sampling of the shows. You could allow users to watch old TV shows on the internet and then build a profile of them and their shows. Much how Amazon works, you could then suggest other shows, other DVDs of shows or perhaps build a site that randomly shows the user episodes that they might like based on prior viewings and statistics of other users.

The take away from this article for me was the fact that Google has vested interest in archiving and now television will be archived Google style.

I can't think of many other uses for this as the system isn't really "inferring" or "thinking" about data samples but is more so matching extracted features against a database. You know, voice recognition software allows for decent voice fingerprinting. You could most likely easily identify characters based on voices (but not actors due to stars like Hank Azaria who do multiple voices). Then you wouldn't need a database of all shows but more so just a database of character voice fingerprints. I would find this sort of approach more interesting but less specific and useful.

Aside from showing this off to your friends, it's not very useful. What I personally would like to see this new Google strategy applied to is all the tapes recorded of famous people like the United States Presidents. If you divided those up into sessions and I was listening to a particular tape of the Nixon set where he talked about the "new right", perhaps a database with references would then point me to some tapes or materials on Joe McCarthey's staunch views on the right.

What shoes did "x" character wear on "x" shoe last week. And, where can I get them? A difference, but analogous in the sense that finding lines from shows is pretty difficult, even for those which are pretty well known. I like the profiling aspect you point to, perhaps another target is "real" measurement. If 15% of people watch a certain show, but 25% of the internet is talking about it, which is a more valuable television property?

Designed to maximize user privacy while minimizing dependency on unique hardware, the system described in the paper seems interesting and feasible. In order to protect user privacy, the software uses "summary statistics" automatically generated from ambient audio rather than transmitting an actual recording. The actual audio cannot be extrapolated from the summary statistic data, so the system doesn't "overhear" or transmit user conversations.

Still, if the data reveals what show the person is watching, your President or anyone else who gets to see the data might start treating you differently depending on what you are watching latley.

So Google researchers find a way to find you TV watching habits with only a simple computer microphone, and in the same paper they describe how they could use the microphone to find more about you for your online profile?

I was thinking exactly the same thing. I wonder if eventually we'll see google toolbar unnoticebly turning on microphone to listen in.Figuring out TV ratings is an expensive things for corps. Google can get a huge market by automating the job of figuring out whose watching what at any given time---if they can only convince the users to let them (``hey, install this google audio-ad analyzer and get 10 gigs added to your gmail account---and it lets you see tv listings in real time, as well as clips from tv pr

If I had a sound-removing tool for TV, I'd block out the laugh tracks. I recently realized that the only network shows I watch, My Name Is Earl and The Office, don't have laugh tracks. I just can't bare to watch the majority of sitcoms that do.

I agree - the laugh tracks are a huge barrier to watching. But a show written around punchlines and laughs is fundamentally different from The Office. Sitcoms with the laffs stripped out would probably appear empty and slow-paced.

in one incarnation i beleive they included a jack on the device and the end user was suppose to hook up an audio cable from one's tv to cuecat v.2. the computer would do all the heavy lifting, eventually finding a hidden tone that would magically pull up an advertiser's web page.

Yeah, I thought I was the only one who realized Google was reinventing the Cuecat, just making it a bit smarter. I don't see that this has a ton to offer to the end user, but a TON to Neilsen and the networks.

I have not owned a TV for 17 years, so Google subcontracting or farming out to their own sub-conglomerate-company 'researching' the use patterns of TV viewers is in no way going to directly affect me getting spam calls to find out what I'm watching, and then sell me more spam calls or directed advertising on based on use. Is Google really that determined to become the next M$, just as invasive and just as annoying? I had hopes they were not.

well not quite the only guy in the US without a TV. I didn't expect you or Google to really give a "crap" one way or the other what I think, however that fact or your opinion certainly doesn't discount or negate what I said.

Not sure about PPM's tech, but Nielsen's A/P meter does exactly what TFA describes. That's the only way Nielsen Media could roll out Time Shifted Viewing [slashdot.org] at all (disclosure: I work for them). To say that Google "created" it is an insult to the people I work with every day.

I see a patent suit in Google's future. As much as I hate patents and like Google, I'd like to at least see some full disclosure here. To (erroneously) state one one

How about outlawing electronic easedropping without written consent? I won't use Macromedia Flash because it turns the microphone on. That's creepy and all non free software with a microphone can do the same thing. It would be better if that kind of thing were against the law.

In the mean time, I avoid non free software and even have bad thoughts about my cell phone.

Joking aside: imagine this as a legit non-marketing use. While watching the news, GoogleSnoop detects some lies being spun by some pundits, and immediatley provides links to opposing viewpoints, definitions, articles of law, or any other silly "facts" the reporter hasn't bothered to research.

Even from a marketing point of view, imagine if a commercial for "Product X" came on, and suddenly Froogle launched and found the cheapest 10 spots online to go buy it.

I don't watch TV much, so I could care less about identifying the TV shows. But what I really would like is an app that would accurately identify mp3 files and apply artist, track #, ect. I've tried a few of the availible programs such as Replay Music and their accuracy is horrid. Maybe Google can do it better. Of course the other use I see for this is identifying music in movies and older TV shows. Newer TV shows do a great job of identifying music, but some older shows (season 1 of The Wire) have great music clips that aren't named in the credits.

MusicBrainz is the service you're looking for; it's been around for a few years now and they're just coming out with a new beta for Linux and Windows users alike. Someone already mentioned iEatBrainz, the Mac variant.

I'm wondering if a similar system can be used to cut out commercials from TV shows? It would be great to have some system that doesn't rely on anything incredibly sophisticated to accomplish this. Perhaps like tripping on commercial type keywords and then using IR remote to mute the TV or switch it to an usused video input to blank the screen.

to be honest i think i would find watching 5 minutes of blank screen just as annoying as the ads,
here in the UK they used to transmit a little grey square at the corner of the screen when the ads were about to come on and you used to be able to get VHSs that could skip the ads that way but I havent noticed it in the alst few years so i dont know if they still work

Rather than "improving" the content of the commercials I see, how about using the technology to recognize and mute commercials that I've previously flagged as either really, really annoying (eg. that oil company's talking cars, which is like being stuck on the subway listening to really stupid people talk about stupid things), or way too loud relative to the actual TV show, or simply shown too many times?I, for one, would gladly pay $10 extra per month to have a button on my remote that when pressed kills t

This very statement presupposes that other noise is irrelevant, which seems bogus.
Snoring is background noise, and suggests non-watching.
Laughter is background noise, and suggests careful watching.
Of course, the laughter might not be about what's on TV...

watch [reference.com] v. tr. 1. To look at steadily; observe, carefully or continuously: watch a parade.look [reference.com] v. To employ one's sight, especially in a given direction or on a given object:
--The American Heritage (R) Dictionary

It seems to me that watching is an activity involving the eyes and mental processing.
It seems to me that audio of what is coming out of the TV is not a statement about either the eyes or about mental
processing. This technology of Google's may be an advance in something, but I hope the advertisers paying for this data have their eyes open about the nature of what they are buying because (to re-mix a metaphor) to my eyes this sounds a bit suspect.

Sociologically, it sounds like a foot in the door to get harmless censors in place. Oops, Freudian slip there. That's sensors, I mean. Google would never involve itself with censorship.

Once the sensors are in place, when "we" realize that it's not getting "us" the data "we" want, we'll just do a few "harmless" downloads of "upgrades", perhaps causing a minor tweak to look at the video data rather than the audio, or perhaps doing language processing after all, and... With user-friendly software like this, who needs spyware?

I also question the claim that because no information is transmitted back to Google that this is the definition of not invading privacy. How is this fundamentally different than the claim that if the police search your house but find nothing, they have not invaded your privacy because they've not placed any record of illegal activity on your permanent record?

It seems to me that once you place a Turing Machine into someone's environment, capable of doing arbitrary processing, and all it sends is a sanitized report, you have all the mechanism in place for abuse. What if the Turing Machine, capable of arbitrary processing, decides that it doesn't want to send a sanitized report. Who is auditing what is sanitized and what is not?

What if it turns out to later be possible to lift information from the supposedly cleansed records? Who will audit the use of that data?

And my mod points just ran out. Too bad.As said, so initially the mic's are exactly what google promises, there are too many concerns with putting in a device conveniently designed to listen, record, and analyze incoming conversation. This is a lot of power to give someone.

How much do other people -really- value your privacy? Would you listen in, just for a moment, if you could? It's tempting just for curiousity.

Add a payment for every person who says they like Happy Margarine. Intellectual interest: H

I'd hate google desktop (or any other google utilitty) spying on my mic to discover my musical preft or anyting else. no tv in my home, but what about the speed at which i type or the general noise in my home or how often my phone goes off or how hard or long my baby cries.. do not listen on my mic, please: 'click' . imagine how many things can be recorded and easily recognized in a home. and many a pc/laptop/headset has a builtin mic, useful to skype, which can thus be used. horror.

I'd like to implement something like this for myself, but with conversational noise instead of TV. I sometimes use my laptop as a visual aid during conversations in my living room. If we're talking about a particular topic, I may pull up a relevant wikipedia article, or something like that. I wouldn't mind if this were more automated.

I can envision running a speech-to-text translator on my laptop mic and then piping that text into my beagle desktop searcher, or maybe even one of those google desktop search tools on windows. I'd rather not send this data to google, for privacy reasons, though.

I could see this being useful at work, or in a conference or class, too. I could stand to have relevant pieces of notes that I took from previous classes pulled up with my professor mentions a particular topic.

Alright, the article says that you need a microphone listening to your TV right? Now, unless Google is sneaking microphones into everyone's homes, the only way you could be spied on is if you agreed to have one placed in your home. If you agree to have one placed in your home, you probably aren't worried about Google spying on you.Saying that Google will use this to spy on people is like saying that the NSA will spy on people who email them all of their personal information, daily habits, etc.

Unless of course you -have- a micrphone already, and install something like google toolbar (or have it pre-installed by Dell, or something). The point is that they wouldn't do this -covertly- (do no evil?), but if they -wanted- to do it (and get clickthrough agreement from the user) they could.

And if all of a sudden they turned into a spyware corp (not that far off their business model), they may even do it covertly---but that would be the end of Google.

I think this sounds like a great new tool for enhansing television as a learning tool. Shows like NOW [pbs.org] are often telling viewers to log onto the website for additional information. Wouldn't it be nice to simply have the computer pull that info up for you as you watch the program? This could be top down or bottom up, with tv shows creating web pages to be loaded, or a wiki style approach where the viewers themselves suggested links(this would of course be difficult during first broadcasts).

Is this something new? At least 3 years ago I was using www.yes.net to get the name of a song on the radio. They've changed their homepage since, probably to commercialize it more, but I used to input my city, the radio station, and the time of day. Their site would display the song that played and offer a link to purchase the disc.

I didn't even RTFA, but from the summary I have an idea on how to implement this idea, it's fairly simple, although it's probably not as computationally efficient as what they came up with, no need to be a great engineer, if you have studied digital signal processing for a few monthes it will be enough.

So you take that audio clip, and you simply cross-correlate (reverse in the time-domain and convolve) it with your audio data base. The highest peak in your results denote a correlation between the audio clip

There was a paper at Webmedia 2005 describing a system -- deployed in early 2005 -- used for real-time audio finger printing that does the same, AFAIK.

http://doi.acm.org/10.1145/1114223.1114238 [acm.org]
This paper describes a scalable real-time audio fingerprinting system developed by IBOPE Midia for radio and TV broadcast monitoring. A special temporal feature extraction strategy based on the Short-Time Fourier Transform has been designed. When given an input stream to analyse, the system matches it against the