Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

Reality? It affected everyone who has automatic updates on mcafee for enterprise, which roughly translates to a large majority of enterprise customers. Usually from a security perspective it's seen as bad form to not have updates available as soon as possible.

It also shows that mcafee's quality control is nothing short of crap. It's known that viruses do rename as svchost sometimes, but clearly they didn't test the heuristics here.

I suspect that after this event, lots of enterprise customers will adopt the stance you propose... either that or they'll abandon McAfee altogether.

The company I work for got hit by this. My personal machine was spared (not running XPSP3), but many, many of my colleagues were down for an entire day or longer while this was getting figured out and cleaned up. A quick back-of-the-envelope calculation for lost productivity at my company alone would easily climb into 7 digits... possibly even 8 digits. Now mu

I would guess there are more than that because of previous licensing. Luckily their licensing ran out on us and we switched to Norton since McAfee hasn't really done much since 2003. There enterprise stuff has really sucked for a while now but we had to wait to get out of the deal with them because of "you know" the economy.

Well, it depends. How many have their computers set to pull updates hourly? If you pulled the updates daily, and it was released an hour after you checked, you were fine (considering they pulled it the same day). So the only computers affected were those that polled in the several hour window that the update was available (Something like 8 hours IIRC). And that's not to mention those configurations that are set to pull updates weekly or more.

You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.

You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.

Not only this, but many Administrators manually review virus' before they are cleaned. I have caught a few false positives by doing manual checks.

We're talking about McAfee running on Windows. Way to be off-topic and ignorant yourself.

That isn't to say Unix and Linux boxes never run anti-virus protection. Some just run on mail servers to protect against virus attachments. But when you run anti-virus in a *nix environment, you often still run real time protection.

It really depends on the intersection of folks running McAfee along with SP 3 in the enterprise. My company is just finishing a migration to Vista, but we still do have about 15,000 Windows XP SP3 desktops (not done deploying yet). However, late last year, I was at a MS Global Accounts meeting (35 very large companies) and NONE of the rest of them had deployed SP 3 for their XP machines. They were all on SP 2 and were harping on Microsoft about the end of support for SP 2 that was fast approaching. None of

None of them wanted to deploy SP 3. It was flabbergasting to me, but they just didn't want to do it.

Some fucktard in a suit gets told that they don't care about problems caused by not running SP3, running SP3 requires a bunch of money to get spent and if he spends it he doesn't get a new BMW 7 series this year.

Really, so many of these decisions have nothing to do with rationality. At some high level it comes down to some guy in a suit angling for a new car, a new house or some other luxury/status symbol.

He tried to reproduce it and had problems? The summary of the problem made it seem like all svchost.exe's would get deleted no matter what.

I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

We were hit by this but I called the guy who manages the AV server and told him to halt any updates and roll back to 5957. Only about 15 systems were hit with it, but none of them had SVCHOST deleted. I was able to isolate one and it was fine since we didn't have the "scan process" enabled. Here is an e-mail I sent to my department:

1. It was on 5958, but everything was running fine.
2. Since I knew there was a fix, I ran an on-demand scan.
3. McAfee picked up SVCHOST.EXE as a virus, and it tried to delete it but the clean failed.
4. Since the clean failed, all I had to do was manually run SVCHOST.EXE from the command line, force an update by right-clicking on the McAfee icon in the systray, and reboot. I ran another memory scan and there were no red flags.

And for this:

I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

Specific conditions had to be met, but they were broad. The following were necessary:
- Windows XP SP3
- Real-time Scanning Enabled
- Definitions version 5958

Everyone that received the patch running XP SP3, yes. However, where I work, they download the patches in the morning and deploy them later on in the evening. So yes, there is a window of attack there, but it saved us from having to go through every SP3 machine and copying the deleted OS file. Basically, everyone else that gets the patches instantly are 'our' guinea pigs.

ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars.

A chain of supermarkets close down, and they only lose thousands

of dollars? Really? I would expect that figure to be a lot higher than that for a single store... Think about all the fresh produce that'll go bad (that have daily deliveries). Think of the power usage (lights, refrigerators). And that's assuming that they aren't paying any of their employees while the store is closed. I'd imagine the loss would be on the order of tens of thousands of dollars per store. Not thousands of dollars across all of the stores...

I would think the same, but it could be a discount supermarket with really low profit margins on dirt-cheap products from second-rate suppliers. We have a chain like that in our area where they leave out the produce until it gets moldy and then offer a replacement guarantee. So if you're 5-day old fruit turns moldy on you, you can return it, but they don't have to toss out as much because people tend to use the fruit within a day or two of purchase. If this was a reputable supermarket, I could see shorte

Nah - this is Coles. That'd be one of the "big two" Australian grocery retailers, with thousands of stores nationwide. I expect that 'loss of thousands of dollars' was many, many thousands (either that or it only affected a very small number of stores for a very small time before getting fixed).

Actually I used to work at Coles (it was my first job!). Our store was the smallest one in the state but still had revenue of ~$300,000 a day...

Agreed. And that's just the immediate cost. When things like this happen, stores/businesses lose loyal customers to competitors and it takes months to recover.
And what about the IT costs? I guarantee you, there is now an effort underway in all major businesses to (1) test new anti-virus patches before rolling them out, (2) re-review all anti-virus software being used, (3) developing and testing mitigation plans for another failure.
All of this is VERY expensive.
Here's another example: Airlines shut d

At least one of our customers were affected as they run our point of sale software on XP Pro SP3 and used McAffee as their anti-virus. That was the IT environment they chose, we told them we prefer OSX as our first choice/Linux as second choice, but they already had a previous POS solution deployed on Windows.

Yup. Same in the organization I am currently working with. Out of 10s of thousands PCs potentially affected, only ~800 actually got nailed, fortunately none at their retail locations. I was one of the lucky ones. After we determined it was an AV issue I was up and running a few minutes later. Safe mode -> rename/delete the latest.dat files -> reboot. Mine didn't delete the svchost.exe like some others did for some reason. Sucks for the folks that aren't somewhat computer savvy and had to have

...If McAfee has a clause in their EULA somewhere that limits their responsibility, and should that be the case, if it is legally enforcable.

Maybe someone with access to said EULA could look it up?

Microsoft once pushed their accountability as a selling point for the Windows Server platform against Linux, if I recall well -- however their maximum responsibility was something like 50$. I wonder what is McAfee's stance in this regard.

By the time you are dealing with large enterprise customers, you aren't dealing in EULAs anymore, you're dealing in negotiated contracts where the legal department of each company goes over each and every clause in the contract.I was talking with some of our IT folks as this unfolded (as my work machine was one of the ones affected) apparently after we were bitten badly by a vendor bug a few years ago, we re-negotiated with most of our software vendors. Our contracts now include penalty clauses for this sor

Yes, but it is the government who put that stipulation in for the contractor. So I am still maintaining they did something right. Whether or not the contractor actually works for the government or is just contracted is irrelevant. The stipulation is there and is there becasue of the government.

It could only effect that few if the policies were set up update infrequently (ever few days or so). My policies are set to check for updates and push them frequently, so I got bitten. I have less than 100 desktops but am a 1 person shop. 4 hours of sneaker net repairs and corporate downtime. Thanks McAfee. There was at least 1 hospital in the area that had to resort to turning non-critical patients away. Don't these things get testing before release? These products are a necessary evil... they don't

Is that it would only take 1 oil and gas company who usually handles Million Dollar deals. Lets see.International Corporation... Lets say 3000+ Employees... lets say just half the company goes down. Rule of thumb is 1 IT guy for every 100 computers (but we all know thats in a perfect world).So, the simplest way to get out of downtime is to go into safe mode and disable the Antivirus, right? Lets say it takes on average 5 minutes to walk to each machine and preform the steps. 500 minutes, or 8.3repeating hou

I've read a few interviewed accounts where the story was much like this:

We applied the updates, and rebooted, then I went on to kick off the others. When I went back to the first couple of servers, I noticed they had rebooted again... then I knew something was wrong.

I know things can't be 100% perfect in an IT world, and yes, virus definitions can be touchy when sometimes zero-day shit can really cause havoc, but I, myself, have of test boxen on my network that I test all patches/updates/virus definitions on for *NIX and Windows boxen. It's not perfect, because to test and interrogate everything is impossible, but I don't apply things blindly. And yes, I've had a few fallout where the package/pat

i've been using Winders since the mid 1990's along with AV software. I have never seen an issue where a definition update has caused something like this. i've seen plenty of times where you can't run an old version on a new OS or issues with games or some software. but letting something out like this into the wild just shows that there was no testing done just to make sure it's OK

I feel sorry for that super market chain but: wtf is AV doing on a POS computer?

POS should be a dedicated computer, running one and only one application (the POS software), on a thoroughly shielded LAN, talking to only a centralised server (or small network of servers if one is not enough) that collects the sales data and distributes prices etc. That server should itself be connected only to the POS network and a corporate LAN. In other words: no direct access out of the Internet, no web browsing, no local storage of any data files, no downloading, nothing that could have the most remote risk of a virus.

McAfee must have had a really good sales guy to convince a Project manager that the POS machines needed AV, either that or who ever developed the POS machines didn't decide to secure them with Enhanced Write Filter, SteadyState, DeepFreeze or some other disk write protection so every time the machine is rebooted it loses all its write cache.

Even though it is Windows, there is absolutely no need for AV when the application is so limited.

Even though it is Windows, there is absolutely no technical need for AV when the application is so limited.

Fixed that. I am afraid that the Payment Card Industry (PCI) differs from your opinion.* In their infinite wisdom**, PCI has decreed that ALL computers need to be running AV. After, all, if it is good for the desktop, it must be good for the servers, right? And since a virus can be spread from anywhere to anywhere, all computers need to have their own protection.

I know it seems silly, but many of the PCI Audit Drones actually believe this. I spent hours trying to convince an auditor that we did not need AV on a Linux server that cannot accept email and has no internet connection. If the PCI Audit Drone finds a computer without AV, you fail the PCI Audit. If you fail the Audit, you get marked as failing on a public web site. If you fail enough times, you lose your ability to accept credit cards. So the need to have AV on a POS is there, it is just not a technical need.

I agree.
However, when you have 200,000+ POS machines, management wants an AV.
I hate McAfee, I hate using a AV instead of isolating a machine from removable media and the Internet. I hate spending money on AV when we could use it on something else. But when a franchise manager on the other side of the world lets one of his employees use the wifi or a printer or something, I'm glad there's an AV to protect my ass. Even though there shouldn't be a way the POS machines get a virus, the AV is kind of like ca

Since when does insurance protect you from accidents? It only compensates you when an accident happened already. If you want to have a car analogy then you should compare AV with seat belts or air bags, that are prevention measures.

Air bags and seat belts don't protect you from accidents either. But, I think they are a good analogy for AV software. You still have an accident and it still hurts, but you are less hurt and might survive because of it.

Most small businesses that are service related have at least one Point Of Sale machine up front at their physical store, but the person operating it is also the person who makes appointments, so they just about have to be able to bring up a scheduler and appointment manager. A separate terminal for appointments is a serious cost, as would be keeping separate people to operate it, or training across skill sets (your cosmetologist or hair stylist or auto mechanic now needs to be trained to schedule appointmen

Typically the POS desktops are talking directly to a server in the backroom. The server in the backroom is typically where a manager will check their emails (via Outlook), take training via a web site, etc. and it's also where the database for the POS client desktops is stored. Every night that small store server submits the data to a main server at the "home base".
So, if the virus scan is on the server (typically is), and the machine goes down, then the business is effectively closed. It's not that th

You're missing nothing except one minor point: no POS system - or anything else in the chain - should be running Windows. This should be a non-issue. My advice to the Australian grocery chain is to fire whomever in the IT department thought this was a reasonable idea.

and why does a POS computer have an internet connection to get the updates? It reminds me of the story of how a bunch of trains had no signal systems because the computers controlling the railway signals were running Windows, connected to a LAN, and got infected with a virus and stopped operating the signals. I guess with admins, you get what you pay for and maybe those MCSE certs are worthless.

It's required by PCI-DSS. Anything that is touching Credit Card data has to be running AV. Our e-commerce servers run on FreeBSD. Guess what, they're running ClamAV. Not because there are viruses for FreeBSD, but it's a PCI requirement.

It is generally accepted practice that windows systems _require_ av, wether it does much good or not is highly debatable (i do a lot of incident response work - ie identifying the source of a breakin, and every system that i get to investigate has some kind of av installed slowing it down)... Infact, i have often had people complain about linux or mac systems without av installed. It's very hard to fight against "standard practices" even when those practices are blatantly flawed.

First, McAfee blew this big time, that such a bug made it to production shows a complete breakdown in their internal processes. XP with SP3 is the number one OS combination in enterprise environments, and should have been the first thing that they tested on. Without doubt McAfee has liability on this and needs to get aggressive about damage control with clients.

That being said, every one of these clients that was hit by this is just as guilty as McAfee is! They are in no better shape and those responsible need to be going management review for their failure. Enterprise Management 101 - nothing goes into production that has not been tested in a lab for pre-pilot and a small group of production computers for pilot! This is as basic as enterprise management gets. Every single environment that was taken down by this shows professional incompetence by their requisite IT departments.

The only question is if it is the fault of management for failing to allow the budget and support needed for a lab for testing or if it is the fault of the IT staffer who never tested things as they should. This is without doubt one of the most public examples of IT incompetence to make the news in years. This is a case of sheer and utter incompetence by every affected party and no pity should be given. If pity were to be given, give it to the poor desktop techs that have to go around making apologies and manual fixes for everything.

As a matter of fact I do expect that. I have designed and set up processes for patch management, software distribution and similar testing for large enterprise environments for years. I have done so everywhere from very large financial institutions to health-care and government. The fact that you need to test daily does not change any principal of what I have said. For any enterprise not to have a dedicated lab to do exactly this kind of testing, or ever worse, not to to use it is sheer and utter incompetence.

In no case should an automated update for an environment ever be released into production without testing. Even Microsoft gets this point and allows you to disable automatic patching to ensure that proper testing can be conducted. I'm not trying to sound harsh, but in all seriousness if you can't learn why testing/every/ production change is necessary from this debacle, than you do not belong in enterprise management. It really is that simple.

How much is your organizations downtime worth? When you have a computer go down, how much is the downtime for computer per hour? If that computer is in a factory your downtime could easily be in the ten's of thousands of dollars per hour. How much is your downtime for a financial computer worth? How much money does your call center lose per hour for downtime? Perhaps you don't care about how much money your company loses for downtime, but you might care about the workers who can no longer perform their job

A buddy of mine is in IT at a college in the area. This affected almost all of their computers. Although it's harder to put a dollar figure on, the students and professors were NOT happy when all of the computer labs on campus went down, along with a "server" or two. Ever seen professors gets mad ? Now imagine your an IT guy and the professors can't access their online grade books that you pushed them into using. I really think McAfee is going to have a big problem on it's hands come contract renewal time. Pissed off IT people have long memories!

We use Sonicwall's security services, their anti-virus is a crippled version of Mcafee business. And we've been hit hard: Machine where going down but WITHOUT any explanation or any warning messages (this version is silent to the user) and since svchost was killed, no chance of getting in the event monitor or using any tools, it took me couple of hour to figure it was the AV. I am sure they "forgot" to add all those third party security solution who rebrand Mcafee solutions. What is making me mad is the way

"McAfee Interwebs Secrutiny has detected that your outgoing mail to customerservices@mcafee.com, subject "You f**king idiotic t**tballs of a son of a ****** in the ******** with a hatstand!!!!" has been detected as Offensive Spam and will be deleted. Thank you for Trusting in McAfee! [TM]"

On a more serious note, I ran into a few small shops that were badly hit, but most of the people I know who work in the enterprise have a time delay before the updates hit the machines, which is usually a hangover from the last time $av_vendor bollocksed up an update.

Personally, I'm still a believer in most AV's being worse that the viruses themselves, and don't run any on my windows boxes - I don't think I've used a single one that hasn't fucked up at some point. Most of my colleagues feel the same way (and, IMHO, by the time it's hit your filesystem and you have that 20% chance of the AV detecting it, it's already too late anyway) and the only reason we run it at work is because of compliance issues... that and the majority of machines being a poorly patched IE6. Yay!

Here's the thing.. it's not Windows' fault that some random program deletes svchost.exe , just as it isn't Windows' fault that any app or user can delete ntldr (e.g. a badly designed uninstaller).

But it -is- a Windows problem because without those, it won't start up. So why is Windows even allowing these files to be deleted?I can't delete by hiberfil.sys even though all it is, is pre-allocated space for the hibernation functionality. If I deleted it, nothing would be lost, and upon hibernation it could re-allocate the required space or tell the user the drive is too full and they're SOL. But no - I simply can't delete it. But I -can- delete vital system files.

So, no.. it's not Windows' fault that McAfee's virus scanner deleted the file. It -is- Windows' problem that they -can- in the first place.

I realize that sometimes there may be a need for a 3rd party application to modify a system file - however rare - but then provide this through a proper mechanism that backs up the original and deletes/replaces on reboot only, with the option to deny the change on boot-up. ( System Restore points only go so far as you'll need the Windows CD/DVD in order to get to the restore utility if you can't boot into Windows anymore. It's also an overly complex solution to the simple problem of renaming files on bootup. )

a) Windows has serious flaws that exacerbate the problem (only recently did they get something roughly sudo like that is still laughably trivial to bypass, and even then poor third-party implementations that haven't grown out of the Win9x days further torture things), nothing short of disciplined users can do anything to get rid of anti-virus market. So long as a user is actually allowed to execute what they want on a system, some stupid thing will convince them to execute it, and damage/manipulate any dat

> So long as a user is actually allowed to execute what they want on a system

BTW, who even thought that was a good idea? Corporate users get a PC for a purpose, and all required applications should be provided. And even if not, a white list should cover 99% of all required software.

Of course as a user I know that things are not that simple. If the only provided browser is IE6 (actually IE7 since recently), Java, Flash, Acrobat, Quicktime and WinZip are all outdated, and the command line is disabled, then

Try telling that to the PCI-DSS folks (Payment Card Industry, aka if you're running E-Commerce/Point of Sale/anything that touches credit card data). They make running anti-virus part of the requirement REGARDLESS of OS. Running on OSX, Linux, or FreeBSD? Doesn't matter. You still HAVE to run AV software on each terminal that touches credit card data.

Only if the user isn't prone to open random attachments which may be executable (experience shows that that this doesn't hold). So long as he does, then you need AV software on any OS.

a third-party party software does NOT need to know the admin/root password to do its job

This has been the case since Windows NT. The key part that you've missed is "well-writte third-party software". Most Windows software was not, historically, well-written in that respect, largely because the primary platform was Win 9x, which didn't have the notion of user accounts to begin with.

I thought this was the whole point of why the PHB buys the expensive proprietary software vs. the free open software; they want someone to sue. The PHBs of the business world distrust free software, don't understand the motivations behind it's existence. But with Paid For(tm) software, the worker bees need to keep the client happy. And if things explode then they get fired or lose their job, or the company gets sued, and the CEO loses 20% of his bonus that year.