Monday, 5 July 2010

Lies, Damn Lies, and Botnet Size

People like numbers. It's understandable. It helps us relate with difficult to understand concepts.

When we started tracking botnets, one of the big things we often wondered about was the size of the networks. It was a natural question, really. We were very young in the ways of tracking these networks of infected hosts. At the time, there weren't many public numbers. Of course, we'd all hear reports from various security companies that there were tens or hundreds of thousands of hosts infected with virus 'X'. Back then, that's all botnets were to an antivirus vendor -- a series of computers infected with the same piece of malicious code. That these groups of computers were massive networks of systems that could be command to, en masse, carry out a series of commands was incidental.

One of the goals of tracking botnets is to report them to the right organizations for action. This action could be simple shut down of the command and control (C&C) server, or more advanced by working with law enforcement organizations to get criminals placed behind bars. That said, how do we decide how a particular C&C will be dealt with? A simple answer is to return to our earlier discusion and remember that people like numbers, even us. Well, especially us. So from the very beginning, we started tracking the size of botnets by simple counting.

Imagine our surprise, in mid 2007, when we very quickly ended up with some of the largest groups of infected hosts ever before seen. No, not a few of these huge botnets, but hundreds of them. This, of course, gave us pause and we started looking into it. It seems obvious now, but DHCP and NAT have a way of screwing with statistics. A computer may have a new IP address every day. In fact, that computer may have a new IP address every hour. If one simply counts unique IP addresses, a single infected host might be counted 24 times in a day, or 720 times in a thirty day period. Of course, on the other side, NAT can make a botnet seem smaller. Imagine a corporate network of a thousand hosts -- each and every one could be infected, but due to the NAT appear as just one infected IP address.

Sometimes a piece of malware uses a unique identifier. There are advantages to the people running the botnet to have these unique identifiers. Perhaps they find something interesting in a keylog file and they want to get deeper, similar, or more frequent updates from the host that uploaded the particular keylog file. This identifier is randomly generated when the host is first infected and it never changes for as long as that host remains infected with that piece of malware. The smaller, better managed botnets tend to be the ones that use these unique identifiers. It's rare for a really big 'net to use the identifiers mostly due to scalability. If the botnet maintains a database that includes every connection to a C&C server from a particular unique identifier, this can get out of hand very quickly with a large botnet. These unique identifiers are good for us, too. We're potentially able to uniquely identify that host when we see it communicate with the C&C. With these unique identifiers, we can determine the exact size of a botnet.

WAGs and EWAGs

People like big numbers, so the botnets with the unique identifiers aren't the ones that get the headlines (a mistake -- more on this later). This leaves us with simple IP counting, which at best provides Wild Ass Guesses. At this point, we can either give up on botnet size reporting or try and find a better way. In 2007, we added a concept of entropy to our botnet size counts. Put simply, we 'expire' an IP from a botnet after a certain number of days. For instance, if a host connects to a C&C, we count that towards the count of the number of bots in that 'net. After a period of days, we remove that host from the count of the number of bots in the 'net. We refer to that period as 'entropy' and provide graphs using three different values of entropy: 5, 10, or 30 days. Since we don't know for sure the exact size of a botnet where we have to use IP counting, we decided to share the results of our calculations rather than state for every botnet a particular entropy value is correct.

For more information about entropy and to see what it looks like in graph format, visit our botnet stats page.

When we instituted this idea, the entropy concept worked pretty well and lined up reasonably when we compared unique identifier networks to counting the same networks by simple IP counting. It also has passed the 'sniff test' all along the way. Maybe these numbers lowball the real size of a particular botnet, but as long as we're consistent, we should be lowballing every botnet roughly the same amount. Of course, the botnets don't appear quite as sexy because they don't have huge numbers.

Big Botnets

Conficker changed things. This botnet grew very big, very fast. The size was large by any measure, and people rushed to provide the answer to the inevitable question: "how big is it?" Many different numbers were thrown out by many different sources, often differing by multiple orders of magnitude. Once it was understood how Conficker was communicating back with its distributed C&C sites, efforts began to take over Conficker. Success was soon found and hosts infected with Conficker were hijacked to report back to a set of sinkhole servers under the control of the Conficker Working Group (CWG). Not only was Conficker effectively contained, but a large data set was suddenly opened to researchers.

As researchers with access to the CWG data, we started counting the size of Conficker in an attempt to once and for all answer the "how big is it" question. Unfortunately, we quickly found that our typical entropy methodology wasn't going to work -- it just didn't pass the sniff test in this case. As a result, we started reporting on the daily count of IP addresses that were seen as infected hosts. This number actually turned out to be pretty consistent, and showed what appeared to be reasonable growth back in the early of the worm. Over time, the number has remained reasonably consistent.

When one takes a look at the graphs of the collected numbers on the Conficker Working Group site, the original growth and subsequent consistency is obvious. The botnet grew to a peak in November of 2009, and then pretty much stayed the same, or dropped very slightly over time since then. This data leads us to conclude that the daily count of IP addresses for the size of conficker makes sense.

In September of 2009, news was broken about a new large botnet called Mariposa by an information security company named Defence Intelligence. Behind the scenes, Defence Intelligence was working with other security companies and researchers to form the Mariposa Working Group (MWG). According to the Frequently Asked Questions posted on Defence Intelligence's site, the MWG includes "Defence Intelligence, Panda Security, Neustar, Georgia Tech Information Security Center and other security researchers who have asked not to be named." The MWG was able to take control of the Mariposa botnet on December 23.

Once the MWG group has control of the Mariposa botnet, one of obvious next steps is to determine the size and scope of the infection: "how big is it?" The response they came back with was that "At the time of this publication, Mariposa consists of an estimated 12.7 million compromised personal, corporate, government and university computer systems." This was a huge number, certainly the biggest botnet ever! Mariposa was twice the size of Conficker, which is quite a feat. This number, not too surprisingly, spread through media outlets like wildfire. But remember the problems had with estimating the size of Conficker? Just how had the MWG come up with this number? In a blog post by MWG member Panda Security, we got our answer: "We were shocked to find that more than 12 million IP addresses were connecting and sending information to the C&C servers, making Mariposa one of the largest botnets in history." -- unique IP address counting.

Is IP address counting really that wrong? In late March 2010, we reset some of the counters on an analysis we had been performing to get a view of Conficker over 30 days and generated the following graph. The numbers do start from zero, but one can see where the numbers settle by observing that the daily unique IP address count is close to a horizontal line. This in turn lines up with the numbers reported on the CWG site for the daily size of Conficker. The rapidly growing blue line is an illustration of what the reported size of Conficker would be if we followed the same thirty day count used for establishing the size of Mariposa. Put simply, if we used simple 30 day counts, Conficker would have (in a thirty day period) infected 77 million systems. In that same thirty day period, Conficker would have been growing at an average of 2.3 million infected hosts a day. Is this likely? To prove the point, we left the count running. As of July 4 2010, that count is over 131 million. What are the chances of 131 million Conficker infected hosts? Zero. The only conclusion is that simple IP address counting is the wrong way to answer the question "how big is that botnet?"

What Really Matters

Ultimately though, the size of the botnet isn't what matters. What really matters is what those in control of the botnet do with it. According to the FDIC, the bad guys stole 120 million dollars in the third quarter of 2009 via online crime. While they didn't give details as to how that money was pilfered, we do know that various ZeuS botnets have stolen over 12 Million dollars (based on numbers assembled by Aaron Jacobson of Authentify) over the past two years. Those thefts weren't from Conficker, nor were they from Mariposa. Conficker itself has no capability to steal credentials and the miscreants behind Mariposa didn't have time to use the credentials they stole. These two botnets, both large no matter how you size them, only impact the people tasked with cleaning them up. They have had very little real world impact other than to generate a large amount of work for the response.

If one looks at the targets of online crime, it's hard to draw trends but we can make a few educated guesses. In general, they're targets of opportunity. These days, large companies and financial institutions are actually a reasonably high bar for your average online criminal. Reading reports by Brian Krebs, the majority of known and reported business victims of online theft are on the smaller side. There's two reasons for this: 1) they tend not to view information security as a high priority, thus making them easier targets and 2) there's more of them and they simply get caught up in widespread mass campaigns.

Don't get caught up in "which is the biggest botnet". Worry about how the botnet is being used. Worry that it's being used to steal money from mom and pop companies who don't stand a chance.