Further Reading

On a bright April morning in Menlo Park, California, I became an Internet spy.

This was easier than it sounds because I had a willing target. I had partnered with National Public Radio (NPR) tech correspondent Steve Henn for an experiment in Internet surveillance. For one week, while Henn researched a story, he allowed himself to be watched—acting as a stand-in, in effect, for everyone who uses Internet-connected devices. How much of our lives do we really reveal simply by going online?

Henn let me into his Silicon Valley home and ushered me into his office with a cup of coffee. Waiting for me there was the key tool of my new trade: a metal-and-plastic box that resembled nothing more threatening than an unlabeled Wi-Fi router. This was the PwnPlug R2, a piece of professional penetration testing gear designed by Pwnie Express CTO Dave Porcello and his team and on loan to us for this project.

Enlarge/ NPR's Steve Henn in his home office and studio, with the Pwnie Express PwnPlug R2 that collected his Internet traffic for a week.

The box would soon sink its teeth into the Internet traffic from Henn's home computer and smartphone, silently gobbling up every morsel of data and spitting it surreptitiously out of Henn's home network for our later analysis. With its help, we would create a pint-sized version of the Internet surveillance infrastructure used by the National Security Agency. Henn would serve as a proxy for Internet users, Porcello would become our one-man equivalent of the NSA’s Special Source Operations department, and I would become Henn's personal NSA analyst.

As Henn cleared a spot on his desk for the PwnPlug, he joked that it might not provide anything useful for us to analyze. In the year since Edward Snowden pulled back the curtain of secrecy around the NSA’s dragnet surveillance programs, many of the major Internet service providers targeted by the spy agency have publicly announced plans to better protect customers, often through the expanded use of encryption.

Our experiment would answer the question: could a passive observer of Internet traffic still learn much about a target in this post-Snowden world?

Henn dialed up Porcello and put him on speakerphone as we finalized the location and setup of the PwnPlug. As I snapped in an Ethernet cable, Henn turned on his iPhone and connected to the PwnPlug’s Wi-Fi network. Porcello watched remotely as data from Henn's network suddenly poured into a specially configured Pwnie Express server.

“Whoa,” Porcello said. “Yep, there’s Yahoo, NPR... there’s an HTTP request to Google... the phone is checking for an update. Wow, there’s a lot of stuff going on here. It's just thousands and thousands of pages of stuff... Are you sure you’re not opening any apps?”

He checked his phone and found that Mail, Notes, Safari, Maps, Calendar, Messages, Twitter, and Facebook were running in the background—and making connections to the Internet. The Safari Web browser proved the most revealing. Like most people who use the iPhone, Henn had left open dozens of websites; when his phone had connected to the PwnPlug’s network, the browser had refreshed them, revealing movies he was checking out for his kids, a weather report, and research he was doing for work.

In the first two minutes of our test, we had already captured a snapshot of Henn’s recent online life—and the real surveillance hadn't even begun.

Your own personal NSA

While the NSA runs hundreds of surveillance programs, its broad, passive surveillance of the Internet has just two key components: Turbulence, a network monitoring system that skims traffic from the Internet’s fiber-optic backbone, and XKeyscore, an analytics database that processes the captured traffic, using rules that look for specific strings of text or patterns in data (e-mail addresses, phone numbers, file attachments). According to leaked NSA documents and whistleblower testimony, pieces of both Turbulence and XKeyscore are scattered about the world near Internet chokepoints such as the infamous “secret room” at AT&T’s San Francisco offices that has been described by former AT&T employee Mark Klein.

To recreate this setup in miniature, the PwnPlug in Henn’s office was configured as a Wi-Fi access point; it acted as our equivalent of the NSA’s Turbulence. While the PwnPlug is generally used for network penetration testing, Porcello configured the device used in our test only to intercept traffic outbound to or inbound from the Internet, not traffic that began and ended on Henn's home network. The device captured every packet matching these criteria and sent it over a secure SSH connection back to a server at Pwnie Express headquarters in Berlin, Vermont.

Further Reading

How the NSA went from off-the-shelf to a homegrown "Google for packets."

The remote machine at Pwnie acted as our diminutive version of XKeyscore. To emulate the NSA's processing of captured traffic, Porcello ran a number of open source analytics tools against Henn's traffic, including the ngrep packet search tool, the tshark and Wireshark traffic analysis tools, the tcpflow data stream capture tool, the dsniff suite’s passive monitoring tools, and tcpxtract for capturing files within Internet traffic.

For more than a month before the experiment began, Ars Technica and NPR made technical and legal preparations to ensure that any data captured from Henn would be handled with confidentiality and care. The focus would be solely on Henn’s personal online activities; we explicitly did not attempt to penetrate NPR’s corporate network, to hack Henn’s computer or phone, or to grab traffic from Henn's other family members. We would simply watch the traffic passing between our test Wi-Fi network and the Internet in the same way that the NSA collects data from millions of Internet users around the world each day.

Our full access to Henn's activities lasted for several days while he reported a single story. To make Henn as accurate a proxy as possible for the average unsuspecting Internet user, one condition stipulated for the test was that when the PwnPlug was active, Henn wouldn’t take extra measures to avoid surveillance (though he followed his normal operational security protocols). Henn could also pull the plug on our test at any time.

The experiment unfolded in two phases. In the first, we simply observed Henn’s normal Internet traffic. In the second, Henn, Porcello, and I stopped the broad surveillance of Henn and turned our tools on specific traffic created by leading Web applications and services. Here's what we found.

AT&T "brain" updates. Dave Porcello intercepted a file download from AT&T to an iPhone that included default settings for a variety of services. One of those settings, Porcello said, was a switch that tells the iPhone to automatically connect to Wi-Fi access points with the SSID “attwifi”. Attackers who want to put themselves in the middle between a phone and the broader Internet need only have their attacking device advertise with the SSID in the file. That feature can be disabled on iPhone devices, but according to Pwnie Express’ Oliver Weis, that isn’t the case with AT&T Android devices.

I wonder if that applies to unlocked AT&T branded phones?

Edit: I'm glad you did mobile apps, especially with the appification of the internet. People may get a false sense of security because "it's not the web" in their eyes.

Would using a commercial VPN mitigate or eliminate this leaking problem (assuming it is setup properly)? I imagine you would have to use it all the time for everything, but I'm wondering if things like the Google pref would still be easily traceable back to you if you were logged onto Google as opposed to using incognito mode?

Would using a commercial VPN mitigate or eliminate this leaking problem (assuming it is setup properly)? I imagine you would have to use it all the time for everything, but I'm wondering if things like the Google pref would still be easily traceable back to you if you were logged onto Google as opposed to using incognito mode?

I would think using a VPN makes the analysis done in the article much harder. But if you an important enough person the spooks would put considerable effort into identifying your digital foot print.

Such encryption gaps don’t just provide a way to spy on what’s on someone’s phone; they also offer an opportunity for hackers (at the NSA and elsewhere) to attack. Attackers could conceivably build a malicious version of an iOS or Android update or spoof the Google Play store and deliver an “evil” version of an app to a targeted phone—especially if the attackers can also fool the phone into connecting to their own malicious Wi-Fi access point.

Such updates are also cryptographically signed, which is why ordinary users can't generally unlock their devices by simply making an update file that gives them full access (outside of specialized devices like Nexus phones at least). If an attacker has the means to cryptographically sign updates as Google/Apple, then the fact that updates are sent in clear text is the least of your worries.

Maybe it's just me, but the article was a big "no duh" from my perspective.

Perhaps so but, in case you haven't noticed them, there are a lot of people out there that still quite ignorant about such things. When you're done patting your brilliance on its back, maybe you can do a better job at informing them than Sean does.

An important article. Whenever I start considering my own internet use and locking it down, at least a little bit I get overwhelmed. I don't feel the need to tor/encrypt/hide everything, but I'd like to have a better grasp on where my own information leaks are, at least the big ones.

It seems from the number of original bugs you discovered in this limited test that this is not really an area of focus for many of these services. What are the limitations to more widespread testing and securing of these services?

I would definitely like to see a follow up article on what we can do to prevent data leakage. VPN will provide some security but are there other options? Is VPN a perfect solution either?

This covers pretty well the data that is sent 'in the clear,' but what if the attacker was willing to use more aggressive means to obtain data? Can things like man-in-the-middle attacks, malicious payloads, or password cracking reveal more data? What, if any, are the defenses against these?

Slightly off topic but what about communication over cellular networks? How hard is it to listen in on this data (Obviously the NSA has access but can any joe do it too?) What about police departments or private groups using Stingray devices? Are there any defenses against those?

The SSL/TLS handshake would be the most expensive part, but once its set up, a very fast form of symmetric encryption is used, basically if there is lots of little requests it could add a fair bit of overhead, eg every comment we up/down vote is a POST.

Maybe it's just me, but the article was a big "no duh" from my perspective.

Perhaps so but, in case you haven't noticed them, there are a lot of people out there that still quite ignorant about such things. When you're done patting your brilliance on its back, maybe you can do a better job at informing them than Sean does.

Plus, there's a major difference between having a general idea that a lot of your information isn't kept private, and having specific examples of exactly how much information can be gathered about a person. Further, this was all just passive spying, exactly like the NSA does to everyone, which they claim is non intrusive because it's untargeted.

Maybe it's just me, but the article was a big "no duh" from my perspective.

For most of us on Ars, that is likely the reaction. But this was a collaborative piece between Ars and NPR. The NPR version of the story got top billing as an almost eight minute long Morning Edition piece. While NPR listeners are generally better informed than most, security and privacy in the techology is still something that the vast majority of the population is pretty ignorant of. This was a good collaboration, and I'd love to see similar in the future.

Edit: Actually, make that two eight-minute segments, the second one ran today.

Would using a commercial VPN mitigate or eliminate this leaking problem (assuming it is setup properly)? I imagine you would have to use it all the time for everything, but I'm wondering if things like the Google pref would still be easily traceable back to you if you were logged onto Google as opposed to using incognito mode?

In addition to what rockforbrains said, there's also the chance for DNS leaks to occur while connected to a VPN; it can randomly start out of nowhere even with a proper setup as far as I know.

Then there's the matter of logging into accounts, through an email already made previously before connecting to the VPN, and the left over cookies too. There's just so many things that can go awry--so many you'll never fix it all 100%.

Honestly you'd go mad trying to fix it all, I think anyone would, but the effort makes it a thorn in someone's side and is enough for me to enjoy doing it too.

Would using a commercial VPN mitigate or eliminate this leaking problem (assuming it is setup properly)? I imagine you would have to use it all the time for everything, but I'm wondering if things like the Google pref would still be easily traceable back to you if you were logged onto Google as opposed to using incognito mode?

In addition to what rockforbrains said, there's also the chance for DNS leaks to occur while connected to a VPN; it can randomly start out of nowhere even with a proper setup as far as I know.

Then there's the matter of logging into accounts, through an email already made previously before connecting to the VPN, and the left over cookies too. There's just so many things that can go awry--so many you'll never fix it all 100%.

Honestly you'd go mad trying to fix it all, I think anyone would, but the effort makes it a thorn in someone's side and is enough for me to enjoy doing it too.

Also, private browsing modes are pretty much worthless.

I think at this point most would agree that there is no 100% anymore, but like you said, why not make it more difficult. I've been using opendns for years now which I understand is a pretty good idea. I'm curious why private browsing modes are useless? It seems that a lot of the tracking techniques where using cookies of various types. Wouldn't private browsing kill those off so that, session to session, ones identifying marks would be different? Coupled with a VPN that also hides user agent info, browsing would seem to be fairly anonymous.

Phones seem to be a pretty significance weak link though, especially when not connecting over a controlled Wi-Fi, who knows what info is being leaked into the ether.

I'm surprised someone was surprised that commodity VoIP is generally in the clear. I don't even know of any common personal offerings that encrypt the audio streams. I'd bet those phones you see on Ars staffers desks are sending audio in the clear as well.

A trip through wireshark's "decode audio" menu is scary the first few times you try it.

I'm surprised someone was surprised that commodity VoIP is generally in the clear. I don't even know of any common personal offerings that encrypt the audio streams. I'd bet those phones you see on Ars staffers desks are sending audio in the clear as well.

A trip through wireshark's "decode audio" menu is scary the first few times you try it.

A professional VOIP installation would have encryption. But some ad hoc multiuser free conference calling scheme probably would go the extra mile.

Regarding apps, it is not very transparent how much data if any is encrypted. It isn't like a browser where you can see the "lock". Many banking apps in the past were not encrypted.

As a minimal effort pen test, you can set up kismet and wireshark. Fine what channel your wifi is on and park Kismet on that channel. Then use wireshark to sniff the packets.

Not mentioned in the article, but I'd be shocked if the pen test device didn't reveal every device on the LAN. You can literally inventory the computer gear on the LAN by sniffing the wifi. These days it will show your smart TVs and any internet of things device.

Maybe it's just me, but the article was a big "no duh" from my perspective.

For most of us on Ars, that is likely the reaction. But this was a collaborative piece between Ars and NPR. The NPR version of the story got top billing as an almost eight minute long Morning Edition piece. While NPR listeners are generally better informed than most, security and privacy in the techology is still something that the vast majority of the population is pretty ignorant of. This was a good collaboration, and I'd love to see similar in the future.

That's sort of my point though. Most people will just hear that someone eavesdropped on some traffic and be shocked, not understanding that the guy used the spy device as his freaking access point. It's inflammatory to the uninitiated, who can't do anything about it anyway but generate tinfoil hats, and mundane news for the initiated. I imagined gasps from people hearing that the remote guy could see all of the websites the iphone was requesting through the pwn's network.

That's the thing, though - it's one thing for a bunch of neckbeards on a forum to say "no duh" to a story like this. It's another when my mom or my grandma won't use Amazon because the entire session isn't encrypted. I guarantee if I set up a Pwnie device in my office, I'd be capturing traffic right and left from the people that don't have their devices set to connect automatically to the work wi-fi and would instantly think "free wi-fi! Awesome!" These are the people that need to be awakened and start, in their own (l)userish ways, bring pressure to bear on companies that are leaking our data all over the place. I can mitigate my own behavior, but short of being a nut like RMS, too much of it is in the hands of companies that either don't give a shit about our privacy or are exploiting the lack thereof as their primary means of revenue.

The best advice I have is to not worry with keeping the foil smooth and free of wrinkles. You are bound to get some unintended creases. So, go ahead and just wad the foil up and then smooth it flat again using your hand. This hides small flaws in the folding process, gives a pleasing texture, and refracts waves in a way that causes most of them to cancel. Nearly zero emission without all the geometry and BS. Plus, two layers of light foil separated by a thin tissue are more effective than one layer of 'heavy duty' foil.

+ A thing not often mentioned is condensation.Good luck working that out.

Recently my sister-in-law asked me why I was so anal about always clearing my cookie cache, history, etc automatically on browser close. To her it seemed overkill and inconvenient to have to re-login to every website, and so on.

Setting login security aside... I showed her why I auto-clear cookies and exit regularly. Doing a simple search for underwear leaves a digital trail of cookies. It followed me to other sites that had zero to do with the search. There's nothing creepier to me than having that cute bra or pair of panties that I ALREADY BOUGHT (idiots!) show up on a serious news/etc website.

I have nothing to hide per se. There's nothing illegal or wrong with my searches. Having my wife, child, or friend exposed directly to something I bought or searched for while showing them an article/video/whathaveyou is just.. weird.

Maybe it's just me, but the article was a big "no duh" from my perspective.

For most of us on Ars, that is likely the reaction. But this was a collaborative piece between Ars and NPR. The NPR version of the story got top billing as an almost eight minute long Morning Edition piece. While NPR listeners are generally better informed than most, security and privacy in the techology is still something that the vast majority of the population is pretty ignorant of. This was a good collaboration, and I'd love to see similar in the future.

That's sort of my point though. Most people will just hear that someone eavesdropped on some traffic and be shocked, not understanding that the guy used the spy device as his freaking access point. It's inflammatory to the uninitiated, who can't do anything about it anyway but generate tinfoil hats, and mundane news for the initiated. I imagined gasps from people hearing that the remote guy could see all of the websites the iphone was requesting through the pwn's network.

The important takeaway was illustration just how much information we leak.

The article was remotely interesting up until he connected *to the pwn device's network* so it could monitor traffic. I though they were going to show something useful and interesting, like drive-by spying on the average WPA2 network, not the braindead obvious issue that anyone you pass your traffic through can see your traffic.

Now if you'll excuse me, I'll get back to getting downvoted into oblivion.

"Ars tests Internet surveillance—by spying on an NPR reporter"

"A week spent playing NSA reveals just how much data we leak online."

This is the closest to how the big boys do it, which was the point of the article. Your thing is interesting too, but it would be a very different article.

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.