Hi all, this is the last we shall say on this particular matter, else, this could go on indefinitely.
Let us state some facts:

1) It is a fact that it is impossible to guarantee that any system is 100% clean.
2) It is a fact that it is possible to bypass DefenseWalls default protection and download and run malware on the system using standard Windows functionality.

Given these facts, it is also a fact that:

1) Even if DefenseWall had been installed on the “clean” OS to start with, the samples / code / malware used could still have been installed on the system, bypassing Defensewall.

2) When installing DefenseWall on any system, you can never be sure there is not malware on there already, which, with DefenseWalls default settings, will run as trusted.

Given these facts, it is also a fact that:

1) Having the samples installed on the OS to start with / not describing or including an infection vector has made no difference to the validity of the test since they could be placed on the system and run as trusted anyway.

DefenseWall is described by the vendor as:

“the simplest and easiest way to protect yourself from malicious software (spyware, botnets, adware, keyloggers, rootkits, etc.) and identification theft, that can not be stopped by your anti-virus and anti-spyware programs, when you surf the Internet!”

Given the product is described and marketed in this way, it is entirely proper that it should have been included in the test.
As a matter of interest, even if the tests are run as “untrusted”, DefenseWall is still unable to prevent all data being captured.

In conclusion and to repeat:

1) The protection provided by DefenseWall is easily and completely bypassed using standard Windows functionality.

2) Even if we purposefully run the tests as untrusted, Defensewall fails to prevent all data capture and so fails the overall test.
I hope we can now draw a line under this and move on to more productive discussions.

PS. Please do not ask how we bypassed DefenseWall as we will not disclose this publically. We have described the method to the developer and hope he is able to cover this in future releases.

Regards,
Sveta

Click to expand...

well, maybe its just me, but i install my security software as my FIRST installation after a fresh OS installation, so id pretty much 100% say, my systems before i install my security are clean. (so, not impossible)

im sure, there are many others who install their security as their first installation after a fresh OS.

if DW says it needs to be installed on a clean system, why test it differently?

also, your stating that DW is easily bypassed but wont state how, what is the developers thoughts on this?

Sveta, simply for the sake of transparency, can you please clarify if any of the vendors examined in the Online Banking Browser Security Test (March, 2010) has or had a financial relationship of any kind with the Malware Research Group?

Personally, I do not believe that the existence of a financial relationship between a vendor and a testing organization necessarily implies that the results are illegitimate. Yet, it is always wise, in my opinion, to disclose any such potential conflicts of interest.

Thank you.

P.S.: I hope you do not interpret my question as “confrontational” or “offensive” -- neither is intended.

Click to expand...

If you are asking me did someone commission us to conduct this test, the answer is absolutely no. This is an official test and as such, nobody outside MRG knew about it.

You asked me how we are funded. We conduct private testing, analysis and research for numerous vendors. Private tests are not made public and their use is for analysing and improving products only.

If anyone wants to use our tests in any way, they must first purchase a license.

We reserve the right to maintain client confidentiality and therefore will not disclose client names are without their express permission, nor will we disclose fees.

well, maybe its just me, but i install my security software as my FIRST installation after a fresh OS installation, so id pretty much 100% say, my systems before i install my security are clean. (so, not impossible)

im sure, there are many others who install their security as their first installation after a fresh OS.

if DW says it needs to be installed on a clean system, why test it differently?

also, your stating that DW is easily bypassed but wont state how, what is the developers thoughts on this?

Click to expand...

Installing security applications first is not a common practice by many people.

You also have to take into consideration that most people stick to their systems for a long time and install applications on it, so in 99% of the cases you can't guarantee that the system is clean.

I can run as many as possible, ranging from new Zero-day samples, to ones a few weeks, months, it doesn't matter to Prevx, hardly anything gets through Prevx, and this isnt even running highest settings. (High/Med/Med)

according to HMP with all its engines, this was my remaining infections:

Im sure you guys are just right clicking a folder of believed-to-be-infected files and cleaning the machine, seeing which files are left.

* & just because everyone else does, i ran MalwareBytes just to be sure.

So, i decide to find that one missing file (copy>paste to desktop) , and perform a simple right-click scan. (this is after a full advanced system scan - which completed and told me clean/protected)

and now i get this:

I am not sure why Prevx acted like this on that last file, but are we now seeing Prevx perform 100% in one of my own little tests, never seen that before on my machine.

*************

Ok, We are not seeing Prevx get a 100% on my machine, here is the results of that 'delayed detection' on that one remaining file.

Its a screen that ive personally never seen before, and its nice to see. (strange i feel like that really, but i think its good that if the software fails on one file, that this process is not left un-attended )

of course, this has happened during my some-what frequent tests of my software, and this was the only file that Prevx has left through, so VERY VERY impressed.

Note:
%System% is a variable that refers to the System folder. By default, this is C:\Windows\System (Windows 95/98/Me), C:\Winnt\System32 (Windows NT/2000), or C:\Windows\System32 (Windows XP).
The following file was deleted:
c:\boot.ini

Memory Modifications

There was a new memory page created in the address space of the system processes:

2) Even if we purposefully run the tests as untrusted, Defensewall fails to prevent all data capture and so fails the overall test.
I hope we can now draw a line under this and move on to more productive discussions.

Click to expand...

This is the only thing in your post that's actually of some use. You tested DefenseWall by running the executables as untrusted?
That would be completely different from what you and others implied in previous posts.

Ok, We are not seeing Prevx get a 100% on my machine, here is the results of that 'delayed detection' on that one remaining file.

Its a screen that ive personally never seen before, and its nice to see. (strange i feel like that really, but i think its good that if the software fails on one file, that this process is not left un-attended )

of course, this has happened during my some-what frequent tests of my software, and this was the only file that Prevx has left through, so VERY VERY impressed.

Click to expand...

I'd suspect that if you run another scan, it will likely clean it up. Detection for your sample was added purely automatically after it was seen to bypass Prevx initially on your PC - that screen prevents cleanup from running around and instead tells the user that they will likely need assistance. However, in your case, it looks like detection was added after one scan which triggered the cleanup message erroneously: if we had run a normal cleanup after, it would have likely cleaned it up without a problem.

However, it's definitely worth bringing that screen to public view more so than it has been It is a relatively rare screen to see, but it is how we ensure that our customers receive the best experience. If we see that a sample was not successfully cleaned on the first round, we will immediately send them to that page and tell them to contact our support directly - even if we would clean it on the next scan, it is generally better to have us intervene in case there is an improvement that we can make

Installing security applications first is not a common practice by many people.

You also have to take into consideration that most people stick to their systems for a long time and install applications on it, so in 99% of the cases you can't guarantee that the system is clean.

Click to expand...

Sveta has been responding to most of these posts, but I'd like to echo this one in particular. This is precisely the case we see with our users - while it is definitely an ideal situation if you can get a user to install their security (whether it is DefenseWall, Prevx, or any other security product) immediately after installing their OS and then keep it up-to-date, this is an extremely rare case, albeit a common one amongst forums like Wilders.

Users "in the wild" have any number of bizarre pieces of malware already on their PC - while it is still useful to install a security product on top of that, the test which MRG has performed is looking to see the effectiveness of a security product with pre-existing infections, the most likely case for an infection that would steal banking details.

Even if an infection is not pre-existing, in the case of sandboxing applications, the sandbox would have to cover every application all the time and additionally cover every possible point in kernel mode... which is not possible. So, as long as you have an application which can access kernel mode or have one which enters the system through a non-sandboxed program, you will be susceptible to these types of threats. That is where SafeOnline and other products step in - if a threat already exists on the PC before these products are installed, these products try and circumvent the threat while banking online.

Granted, this is not perfect and indeed there is no silver bullet in any of these solutions - SafeOnline is built on top of Prevx 3.0 and many of the other products require the user to use an up-to-date AV solution.

Although it may seem that I am biased because of how SafeOnline scored, I stand behind the methodology that MRG have used as it is valid for testing threats in these conditions.

Granted, this is not perfect and indeed there is no silver bullet in any of these solutions - SafeOnline is built on top of Prevx 3.0 and many of the other products require the user to use an up-to-date AV solution.

Click to expand...

This is precisely the reason that the MRG test is flawed. SafeOnline has been developed in order to fill a gap left by other security products (including Prevx itself). It's not valid to compare SafeOnline against products that have been designed for an entirely different purpose and which have different operating parameters.

Before SafeOnline was developed, had Prevx anti-malware been included in this test, it might have fared very badly. The methodology used for the test correctly ignored the capability of the products to detect, prevent, and clean malware; as the purpose of the test was to test the ability of the products to protect the browser on an infected system. In this scenario, Prevx would likely have been the first to point out that the methodology was flawed and that Prevx anti-malware must be tested as a whole, including its ability to detect, prevent, and clean; that nothing is perfect and a layered defense is always best, etc, etc.

You just can't take programs using different approaches such as policy restriction, virtualisation, whitelisting, blacklisting, heuristics, HIPS, etc and start comparing them willy nilly with scant regard for what each program does, how it does it, and the range of scenarios that determine and limit its proper use.

You just can't take programs using different approaches such as policy restriction, virtualisation, whitelisting, blacklisting, heuristics, HIPS, etc and start comparing them willy nilly with scant regard for what each program does, how it does it, and the range of scenarios that determine and limit its proper use.

Click to expand...

The issues of 'installing the software on a clean system' aside, all of the software tested has specific anti-logger capabilities. The approaches may be different (e.g. HIPS, policy restriction etc) but they all have specific functionality that can be classed as anti-logging. So yes, you can compare the anti-logging capabilities of such programs.

So yes, you can compare the anti-logging capabilities of such programs.

Click to expand...

I take your point but the test was described as an "Online Banking Browser Security Test", not just a generic anti-logging test. Even Prevx anti-malware (minus SafeOnline) on its own has some anti-logging capability.

Of the programs tested, only Prevx SafeOnline and Trusteer Rapport have been specifically designed for the express purpose of browser security on an infected system. Both programs put a wrapper around the browser, operating as a kind of reverse sandbox. IMHO this puts these two programs in a different class to the other programs tested, also witnessed by the fact that both are aiming at gaining acceptance by the banks. In this respect they are direct competitors in a fairly new genre in which there are as yet few players. It didn't surprise me that SafeOnline did so well in this test, but it did surprise me that Rapport did so badly, given that the test methodology was tailor-made for these two programs to shine. For anybody considering deploying online banking security software, it's useful to know how these two programs stack up against each other.

I'm not looking to start a fight, just expressing the view that the test wasn't a level playing field; and I'll say it again - kudos to Prevx for achieving such a good test result, and kudos to MRG for carrying out the test.

Installing security applications first is not a common practice by many people.

Click to expand...

He's totally correct, so conducting a test in this manner is more than just appropriate, it is a Real World test. Of course no one can possibly predict exactly what nasties ALL those people out there have already been infected with, but even so it's a very worthwhile exercise, and i wish there where more like it.

He's totally correct, so conducting a test in this manner is more than just appropriate, it is a Real World test. Of course no one can possibly predict exactly what nasties ALL those people out there have already been infected with, but even so it's a very worthwhile exercise, and i wish there where more like it.

Click to expand...

I agree with the way the test was done also. If the test was about installing on a clean system and then running the malware and using the anti-malware apps as intended, Defensewall and others would have done a lot better, but as the test was about a real world scenario the tests do portray exactly what would be the real world outcome.

Before SafeOnline was developed, had Prevx anti-malware been included in this test, it might have fared very badly. The methodology used for the test correctly ignored the capability of the products to detect, prevent, and clean malware; as the purpose of the test was to test the ability of the products to protect the browser on an infected system.

Click to expand...

I think it is crucial to ignore the detection aspects of these products in this test, especially when testing leaktests. It is very easy to detect a single sample, or even a class of samples by behavior, but what happens when the malware authors test their creations against your product until it doesn't detect it any more? It would likely take less than an hour to make a known piece of malware completely undetectable from every AV-style product, so I think taking the detection of files out of the picture is the only valid way to test this type of protection.

Sorry Sveta, Either they were already on the system or Defensewall was bypassed, not both, which is it?

Click to expand...

That's not what Sveta is saying. He is saying that independently of these tests MRG have found a bypass to DW. Hence the argument that DW would have prevented the system being infected in the first place doesn't hold water either.

That's not what Sveta is saying. He is saying that independently of these tests MRG have found a bypass to DW. Hence the argument that DW would have prevented the system being infected in the first place doesn't hold water either.