Is your refrigerator really part of a massive spam-sending botnet?

Security researchers have published a report that Ars is having a tough time swallowing, despite considerable effort chewing—a botnet of more than 100,000 smart TVs, home networking routers, and other Internet-connected consumer devices that recently took part in sending 750,000 malicious e-mails over a two-week period.

The "thingbots," as Sunnyvale, California-based Proofpoint dubbed them in a press release issued Thursday, were compromised by exploiting default administration passwords that hadn't been changed and other misconfigurations. A Proofpoint official told Ars the attackers were also able to commandeer devices running older versions of the Linux operating system by exploiting critical software bugs. The 100,000 hacked consumer gadgets were then corralled into a botnet that also included infected PCs, and they were then used in a global campaign involving more than 750,000 spam and phishing messages. The report continued:

The attack that Proofpoint observed and profiled occurred between December 23, 2013 and January 6, 2014 and featured waves of malicious email, typically sent in bursts of 100,000, three times per day, targeting Enterprises and individuals worldwide. More than 25 percent of the volume was sent by things that were not conventional laptops, desktop computers or mobile devices; instead, the emails were sent by everyday consumer gadgets such as compromised home-networking routers, connected multi-media centers, televisions and at least one refrigerator. No more than 10 emails were initiated from any single IP address, making the attack difficult to block based on location – and in many cases, the devices had not been subject to a sophisticated compromise; instead, misconfiguration and the use of default passwords left the devices completely exposed on public networks, available for takeover and use.

The Proofpoint report quickly went viral, with many mainstream news outlets breathlessly reporting the findings. The interest is understandable. The finding of a sophisticated spam network running on 100,000 compromised smart devices is extraordinary, if not unprecedented. And while the engineering effort required to pull off such a feat would be considerable, the botnet Proofpoint describes is possible. After all, many Internet-connected devices run on Linux versions that accept outside connections over telnet, SSH, and Web interfaces.

Where's the smoking gun?

Still, there's a significant lack of technical detail for a report with such an extraordinary finding. Among other things, Proofpoint provided no details about the software the researchers say compromised the devices; it said it didn't "sinkhole" or otherwise monitor any of the command-and-control servers that would have been necessary to coordinate botnet activities; and it didn't convincingly explain how it arrived at the determination that 100,000 smart devices were commandeered. My doubts lingered even after a one-on-one interview with David Knight, general manager of Proofpoint's information security division.

Knight said Proofpoint knows appliances sent the spam directly because researchers scanned the IP addresses that sent the malicious e-mails and received responses from the Internet interfaces of name-brand devices. I pointed out that many home networks have dozens of devices connected to them. How, I asked, did researchers determine that spam was sent by, say, an infected refrigerator? Isn't it possible that a home network with a misconfigured smart device might also have an infected Windows XP laptop that was churning out the malicious e-mails?

Knight's response: in some cases, the researchers directly queried the smart devices on IP addresses that sent spam and observed that the appliances were equipped with the Simple Mail Transfer Protocol or similar capabilities that caused them to send spam. In other cases, the researchers determined the devices were connected directly to the Internet rather than through a router, making them the only possible source of the spam that came from that IP address.

Again, what Proofpoint is reporting is plausible, but it doesn't add up. Experienced botnet researchers know that estimating the number of infected machines is a vexingly imprecise endeavor. No technique is perfect, but the scanning of public IP addresses is particularly problematic. Among other things, the intricacies of network address translation mean that the IP address footprint of a home router will be the same as the PC, smart TV, and thermostat connected to the same network.

It's also hard to understand why someone would go to all the trouble of infecting a smart device and then use it to send just 10 spam messages. Traditional spam botnets will push infected PCs to send as many messages as its resources allow. The botnet reported by Proofpoint requires too much effort and not enough reward.

None of this is to say that the reported 100,000-strong smart-device botnet doesn't exist. And as most students of logic accept, it's not feasible to prove a negative. Still, the lack of evidence documenting any malware sample or a command and control server should give any reporter pause before repeating such an extraordinary claim. The research methodology is also a red flag.

I contacted Paul Royal, a research scientist at Georgia Tech who specializes in network and system security, and I asked for his take on the Proofpoint report and the additional information provided by Knight. He was skeptical, too.

"The aggregate of the information doesn't paint an adequately compelling picture that what they're asserting occurred actually occurred," Royal said. "When you ask something as simple as how do you know the spam came from gadgets they say: 'Well, we looked at the IP addresses of the systems sending the spam and when we presumably probed them we observed that they were coming from set-top-box-like devices.' The technical analysis of that shows that there could be plenty of other explanations."

Knight said he would check to see if missing evidence—including a malware sample, documentation of a command-and-control server, and samples of the spam and phishing messages—are available for publication. Again, I'm open to the possibility the botnet reported by Proofpoint exists. But until these smoking guns are produced, I'm maintaining a healthy amount of skepticism.

Yet another article that shows Linux is really not safer than Windows. Neither OS is safer if its not updated. This is for all those that tag Windows as insecure, it happens that most Windows users compromised also run outdated software without the patches. XP anyone?

It would not surprise me that in the last few years, more Linux machines where hacked than Windows boxes because Linux is just everywhere, from servers to phones. Lets not forget the Linux kernel receives an averages of at least 2 security patches per month, this on average.

Now, as we see more devices that are smart, I think this problem is going to be a real problem in the future, because more refrigerators or smart appliances are never going to be updated after they where sold which is a nasty problem in our hands, unless manufacturers plan to update them remotely I don´t see users updating their TVs or microwaves once a month.

The scenario that really concerns me is the "connected TV". It will typically have a browser, and many have support for Facebook, Twitter and mail. And typically no protection against infection.

This is a Very Bad Thing just waiting to happen.

I guess you could always just unplug it from the network. Ohh wait, I assume this are Wifi, which means most will not even know its connected in their homes. You can´t see it so you assume its not there.

I think devices like this should have a hardware switch to turn it off from the network by design. Some laptops have a Wifi switch. This at least will bring users attention to "what does this button do, and he will turn it off if he does not want his refrigerators to be connected" it also acts a fast measure if his refrigerator is doing something strange on the Internet. Simple but effective.

The bad part I see with this, is that some people are not even going to be aware their devices are connected, as with DHCP in their home networks its going to connect directly without the user interaction.

I really don't see the point of connecting a refrigerator, dryer, etc to the internet.

In a lot of case, yes, it's quite a stretch. Most of the "Internet-enabled" appliances now are wasting the potential by putting things like Twitter access on it -- who cares?

But that's not to say that there isn't a potential for some benefit. Maybe your dryer could send you a message when it's done -- useful if you are going to be doing multiple loads, but your dryer is far enough away that you can't hear it. Maybe you can send a message to your stove to start pre-heating on your way home, so that's it's ready to go when you arrive.

Those aren't earthshaking benefits, but they could absolutely be useful.

And then there's the holy grail of a smart refrigerator. You add a scanner on the door and an inventory engine. It always knows what you have, and how old everything is. When you start running low, it can automatically add that item to your shopping list -- synced to your phone, of course. If you happen to be at the grocery store and can't remember if you need milk or not, you just query your fridge from the store. I think there's pretty significant benefit for fridges, if done right.

Look, I know that the very idea of a smart appliance seems silly. But just get a Nest thermostat and see if the previous ludicrous idea of a connected thermostat doesn't start making a lot of sense.

The bad part I see with this, is that some people are not even going to be aware their devices are connected, as with DHCP in their home networks its going to connect directly without the user interaction.

I think a refrigerator that connected to the internet would have that as part of it's marketing material.

But that's not to say that there isn't a potential for some benefit. Maybe your dryer could send you a message when it's done -- useful if you are going to be doing multiple loads, but your dryer is far enough away that you can't hear it. Maybe you can send a message to your stove to start pre-heating on your way home, so that's it's ready to go when you arrive.

I would nix the preheated stove idea. There's a safety/malicious behavior element there. Some things are best having the owner around.

In a lot of case, yes, it's quite a stretch. Most of the "Internet-enabled" appliances now are wasting the potential by putting things like Twitter access on it -- who cares?

@JoeRandom: I just switched to channel 6 on Comcast Xfinity.@JoeRandom: I just switched to channel 7 on Comcast Xfinity.@JoeRandom: I just switched to channel 8 on Comcast Xfinity.@JoeRandom: I just switched to channel 9 on Comcast Xfinity.@JoeRandom: I just switched to channel 10 on Comcast Xfinity.@JoeRandom: I just switched to channel 11 on Comcast Xfinity.@JoeRandom: I just switched to channel 12 on Comcast Xfinity.@JoeRandom: I just switched to channel 13 on Comcast Xfinity.@JoeRandom: I just switched to channel 14 on Comcast Xfinity.@JoeRandom: I just put my motion-aware remote control on the table.@JoeRandom: I just flushed my SmartToilet.@JoeRandom: I just switched to channel 15 on Comcast Xfinity.@JoeRandom: I just switched to channel 16 on Comcast Xfinity.@JoeRandom: I just switched to channel 17 on Comcast Xfinity.@JoeRandom: I just switched to channel 18 on Comcast Xfinity.@JoeRandom: I just switched to channel 19 on Comcast Xfinity.@JoeRandom: I just switched to channel 20 on Comcast Xfinity.@JoeRandom: I just switched to channel 21 on Comcast Xfinity.@JoeRandom: I just switched to channel 22 on Comcast Xfinity.@JoeRandom: I just switched to channel 23 on Comcast Xfinity.@JoeRandom: I just switched to channel 24 on Comcast Xfinity.

In a lot of case, yes, it's quite a stretch. Most of the "Internet-enabled" appliances now are wasting the potential by putting things like Twitter access on it -- who cares?

@JoeRandom: I just switched to channel 6 on Comcast Xfinity.@JoeRandom: I just switched to channel 7 on Comcast Xfinity.@JoeRandom: I just switched to channel 8 on Comcast Xfinity.@JoeRandom: I just switched to channel 9 on Comcast Xfinity.@JoeRandom: I just switched to channel 10 on Comcast Xfinity.@JoeRandom: I just switched to channel 11 on Comcast Xfinity.@JoeRandom: I just switched to channel 12 on Comcast Xfinity.@JoeRandom: I just switched to channel 13 on Comcast Xfinity.@JoeRandom: I just switched to channel 14 on Comcast Xfinity.@JoeRandom: I just put my motion-aware remote control on the table.@JoeRandom: I just flushed my SmartToilet.@JoeRandom: I just switched to channel 15 on Comcast Xfinity.@JoeRandom: I just switched to channel 16 on Comcast Xfinity.@JoeRandom: I just switched to channel 17 on Comcast Xfinity.@JoeRandom: I just switched to channel 18 on Comcast Xfinity.@JoeRandom: I just switched to channel 19 on Comcast Xfinity.@JoeRandom: I just switched to channel 20 on Comcast Xfinity.@JoeRandom: I just switched to channel 21 on Comcast Xfinity.@JoeRandom: I just switched to channel 22 on Comcast Xfinity.@JoeRandom: I just switched to channel 23 on Comcast Xfinity.@JoeRandom: I just switched to channel 24 on Comcast Xfinity.

@JoeRandom: I just switched to the playboy channel on Comcast Xfinity.(JoeRandom's son is so busted.)

Yet another article that shows Linux is really not safer than Windows. Neither OS is safer if its not updated. This is for all those that tag Windows as insecure, it happens that most Windows users compromised also run outdated software without the patches. XP anyone?

The problems seem to be mostly implementation issues like unchanged default passwords coupled with leaving telnet or a web interface open to the outside world. Even a theoretical 100% bug-free OS is "vulnerable" to such misconfigurations. This is vastly different than the myriad Windows XP vulnerabilities in the default configuration.

Same thing that's been said several times. If that were the case, there'd be one device on the network to be updated and secured, instead of...well, however many your house has.

I think home automation is useful--including things like being notified via phone notification--but I don't see why an SNMP-like setup wouldn't be better than full-up networking with Twitter and Facebook. Better, I would think, to have a single home controller for any of that fancy shit.

Knight's response: in some cases, the researchers directly queried the smart devices on IP addresses that sent spam and observed the appliances were equipped with the Simple Mail Transfer Protocol or similar capabilities that caused them to send spam. In other cases, the researchers determined the devices were connected directly to the Internet, rather than through a router, making them the only possible source of the spam that came from that IP address.

That response confuses me a bit. Were some details left out or where they really that vague?

Are they saying that they queried the IP addresses and then determined that the devices at those IP addresses were smart appliances which somehow had an SMTP server running on them, or are they saying that they found IP addresses with SMTP servers, some of which were directly connected to the internet (sigh), and then decided some or all of those devices were smart appliances?

I'm not following the jump to "these were smart appliances" (not that I doubt in the least that smart appliances can, or likely are in some cases compromised by something).

In other cases, the researchers determined the devices were connected directly to the Internet, rather than through a router, making them the only possible source of the spam that came from that IP address.

So someone got a separate broadband subscription for the exclusive use of his fridge? That really stretches credulity.

With a router, it's likewise hard to imagine. Say my fridge uses password "abc". My router doesn't forward telnet requests to the fridge, so what are you going to do with that? It's like with the door: my fridge door doesn't have a lock, yet I don't worry about strangers taking food out of it, because my front door does have a lock. Yes, you can compromise a router, just as you can break my front door: in both cases I have bigger worries than my fridge.

If we're going to have to network our appliances, bathroom fixtures, thermostats, and electric toothbrushes, it's my opinion that all of them should connect to a PC instead of to the Internet. Then if you really can't live without cleaning your electric razor remotely from the office, you're just connecting to the same PC and app that let you lock the cat door and flush the toilet. With only one thing to configure, secure, and maintain, we won't have our whole houses compromised because we patched the ice maker and the sump pump but forgot the garbage disposer.

Knight's response: in some cases, the researchers directly queried the smart devices on IP addresses that sent spam and observed the appliances were equipped with the Simple Mail Transfer Protocol or similar capabilities that caused them to send spam. In other cases, the researchers determined the devices were connected directly to the Internet, rather than through a router, making them the only possible source of the spam that came from that IP address.

That response confuses me a bit. Were some details left out or where they really that vague?

Are they saying that they queried the IP addresses and then determined that the devices at those IP addresses were smart appliances which somehow had an SMTP server running on them, or are they saying that they found IP addresses with SMTP servers, some of which were directly connected to the internet (sigh), and then decided some or all of those devices were smart appliances?

I'm not following the jump to "these were smart appliances" (not that I doubt in the least that smart appliances can, or likely are in some cases compromised by something).

Knight said Proofpoint knows appliances sent the spam directly because researchers scanned the IP addresses that sent the malicious e-mails and received responses from the Internet interfaces of name-brand devices. He also said some of these devices have SMTP servers or other services needed to send spam.

Yet another article that shows Linux is really not safer than Windows. Neither OS is safer if its not updated. This is for all those that tag Windows as insecure, it happens that most Windows users compromised also run outdated software without the patches. XP anyone?

Any machine that isn't setup properly with an eye towards security is unsafe. Manufacturers just like to pretend that they aren't shipping computers tied to appliances. It wouldn't really matter what OS these things run, they'd strip the firewall and security policies out and run everything under the administrative account anyways. Security updates? Hah!

The bad part I see with this, is that some people are not even going to be aware their devices are connected, as with DHCP in their home networks its going to connect directly without the user interaction.

How is my fridge going to connect to my wi-fi without the WPA key?

I can typically see 12-13 wi-fi networks from my apartment. One uses WEP, one is open, and the rest are WPA2. So on a clear day when I can see 13 networks, 12 will not have smart toasters connecting themselves and blasting out spam. Then again, I suppose that poor sucker that leaves his open might have a squadron of smart appliances hooking themselves up and choking his network.

But more to the point, even not terribly sophisticated people aren't likely to have a bunch of appliances setting themselves up on wireless networks.

I'm also not buying ProofPoint's 'directly connected' BS. I just don't see it.

"It's also hard to understand why someone would go to all the trouble of infecting a smart device and then use it to send just 10 spam messages. Traditional spam botnets will push infected PCs to send as many messages as its resources allow. The botnet reported by Proofpoint requires too much effort and not enough reward."

I can confirm the new behaviour has been operating for some months. Well before Christmas we discovered infected clients were sending small batches (10 or less) of spam with varying destinations and random time spacing between sends.

We also catch this incoming spam with 10 or less recipient CC's and some are now even hiding the recipient list so it appears to be a simple email from an associate.

This new stealth behaviour flies under the radar quite well and is hard to catch/block for both sending and receiving ISP. It's much smarter than the hammer, instant discovery and blocked approach that has a much lower reward point.

The scenario that really concerns me is the "connected TV". It will typically have a browser, and many have support for Facebook, Twitter and mail. And typically no protection against infection.

This is a Very Bad Thing just waiting to happen.

AMEN to that. The only thing worse than a smart TV is an obsolete (like last month's model) smart TV with Facebook on it. Guaranteed to be eventually an attack target. Little streaming modules from Roku and others are more likely to be updated with new firmware. Assuming the industrial control marketplace gets its act together and toughens up their Internet access points, the next big botnet resource will be smart TV's and DVD/BluRay boxes with streaming firmware. The scary part is that dozens, or hundreds of models of TV will likely be running the same Linux or Android firmware making it easier to hack more zombies.

My one "smart TV" got his ears bobbed at the firewall, no incoming connections allowed. Wireshark was seeing some stuff that didn't look right, so the outgoing got shut off also. Its a pain in the ass since I have to assign fixed IP addresses to all the devices then lock them out of my LInux firewall. (MAC addresses I've found are random on each bootup) on a couple of my devices. So that's not a way to filter the device.

Please perform the following experiment:For one week, before leaving your office for home, write down the activity you will be doing 70 minutes later.Then 70 minutes later, actually record what you are doing.

In 5 days, you will have 5 misses, but a list of 5 "extenuating circumstances" that prevented your plans from reaching fruition. That's the nature of life.

Now let's add your pre-heating oven to the mix. See the problem? What's the efficiency you are chasing here? Thousands of ovens in homes all over the country burning electricity for meals that will be late or may never be cooked.

Please perform the following experiment:For one week, before leaving your office for home, write down the activity you will be doing 70 minutes later.Then 70 minutes later, actually record what you are doing.

In 5 days, you will have 5 misses, but a list of 5 "extenuating circumstances" that prevented your plans from reaching fruition. That's the nature of life.

Now let's add your pre-heating oven to the mix. See the problem? What's the efficiency you are chasing here? Thousands of ovens in homes all over the country burning electricity for meals that will be late or may never be cooked.

Or worse, something going awry and said oven setting your house on fire with you not there, as Ostracus pointed out.

As I started to read this article, there was an urgent knock at my door, and to my surprise I found an Ars special delivery agent with my Tin Foil Hat that he explained I needed to continue reading the article. Grateful for such excellent service I tipped him and donned my hat and was able to finish reading the article after that.