If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Testing out products - benchmarking software

Hey all

I have been asked to test some software. I have installed a VM and now have to test certain AV products and do a comparison (boot time once installed , scan time etc etc). What is the best procedure? I have created a Vista VM (running VM workstation) and are going to fully update it. I will load no other software.

Will the performance not be accurate because its in a VM on this machine? Must I do anything to my machine as in not do anything while performing tests? Must I disconnect from the network?

Any tips would be appreciated.
Thanks.

The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.Albert Einstein

Disconect it from the network set it up as though it's just a normal desktop PC that grandma and grandpa would use.

Fresh install no VM, and do the test not only on Vista but XP etc. just set it up so it's got bare essentials but is fully updated ie all service packs etc.

and when installing the app just use the normal install method don't custom setup.

That way when testing each one they are all on a level playing field.

Also while testing each one make sure you visit site's with malware etc and see witch one's pick up and witch one's don't. Also disable the av after the first scan and purposly drop a virrii/trojan or something bad onto the computer and re-enable it and do a scan and see witch one's detect it and witch don't.

Place the virrii in the exact location for each app test that way it's all fair.

I agree with t34b4g5's approach re: no VM and disconnected. However, after performing the "out of network" evals, I would place them "in network" for real (your specific env) world experiences.

Also, before you start you might want to ghost your test machine so that you can easily "uninstall" and try a different AV product with the same image. This will remove OS differences as a potential variable. It also might save time and headaches.

1. Do not use VM for testing of this nature. OK for checking updates and fixes, but not for evaluating new stuff.

2. You need "reference machines" that are identical, or a good stopwatch and logging system, and run the tests sequentially.

And another word of advice.............. if you are testing AV products or any other commercial software for that matter............. read the bloody terms and conditions FIRST!!!!!

Standard AUPs generally forbid that you publish anything you find, or even use this other than for your own decision making? If you are working in a commercial environment and use this information to advise customers you will need formal, legal, permission from the owners of the IP rights to even conduct such a test.

I will bet a trillion Rands that your POS employers do not have such permission

Your employers will fire you and make you totally unemployable if it would save them a few thousand rands in legal fees and compensation?

If you cannot do someone any good: don't do them any harm....
As long as you did this to one of these, the least of my little ones............you did it unto Me.
What profiteth a man if he gains the entire World at the expense of his immortal soul?

Its only illegal to publish it I think. If it is for your own company you may be alright as long as you do not make your findings accessible to the public, i.e. on the internet :-p Its too bad vmware cant help you with testing with this nature because snapshots are awesomeee. But you may want to consider to use at LEAST xp restore points that way you can restore after you are done testing that a/v so the reminants do not interfere with the next tests.

Good point on the restore points. This evaluation will be used for sales purposes, it wont be public however the company we are pitching too is quite a large corporation with offices around the globe so yeah I think there might be legal implications.

VMWARE snapshots the next best thing besides Vistas snipping tool.

The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.Albert Einstein

Actually what is probably even better then a built in snapshot is ghost corporate (if you have access to it) You just make a clean install and make a cd with a bootable image of the clean install. Nothing needs to be installed on the system its self and who knows what is left behind after using other snapshot utilities..