Since it is not possible to acquire all s/w and h/w under the sun, we often resort to using virtual machine, user agent, multiple IE etc.

I have had instance when I did not find an application functioning on a virtual machine especially multiple IE but when approached another colleague having the desired configuration, it worked well.

Given such instance I began to doubt using such simulation for testing. Do you also test using such simulation than actual device. Do you trust in testing using such simulation than using actual device?

4 Answers
4

Generally, virtual machines are fine for testing, as they will give the same results as real machines for 99% of functional tests.
However, there are some things that should be tested on a real physical machine before you can declare the test passed:

Drivers, and anything else in kernel space. VMs are completely real to the operating system, but sometimes the abstraction leaks. This is usually only apparent in the very low levels of the operating system, where the VM may not react exactly the same as a physical box. For these cases, at the very low level of the operating system code, you should test on physical machines and not VMs.

CPU Architectures and specialized compilers. VMWare can simulate x86 and x64 compliant CPUs for your VM, but it can't really simulate different processor architectures. In previous projects, I've had to use physical boxes to test a driver on Itanium machines, and to see the differences when using the Intel compiler, which means compiling and running on an Intel CPU. These things are not good for virtualisation, for obvious reasons.

Performance testing and benchmarking for load and stress. In VM setups, the real physical resources are shared between all VMs on the host machine. Even when CPU cycles and RAM are partitioned, network and IO are still shared. This means that you can't really do performance testing, and certainly not benchmarking, on VMs.

The thing is, that even with the above exceptions, where I think VMs cannot be used effectively, I still use virtual machines for 98% of my work in any given year.

Virtualizing test environments is a great way to save on hardware and provide a centralized testing environment, but it's prone to the same types as issues as having a cluster of physical boxes if you don't have a strategy behind it.
In my company we use a few clusters of virtual systems that all derive from a common template, so the default states are as homogeneous as they can get. For each system we create, we tend to take a snapshot of the default state to revert to if it gets dirtied by heavy use. This reduces reset time to a few seconds as opposed to upwards of an hour for re-imaging a physical machine.
In addition, this approach lends itself well to creating server environments for web site / application testing as well as desktop software testing, for running a suite of automated tests and providing automated acceptance tests.
To date, we have never encountered any bugs or behavior that is related to the virtual platform; they still manage to be machine specific inside the virtual test environment.

so you find a defect with app on VM, do you report it a it is or also state that it was encountered on VM?
–
TarunMay 11 '11 at 5:00

Nope. We just report it. The dev mostly work on VMs too. And we did not encounter any defects yet stating: Only on VM. Or only on Live environment. Hack, most of the build machines are VMs. :) There are really only few dedicated machines. And we are a big company in the US.
–
HannibalMay 11 '11 at 5:01

Thanks for your comments +1, I guess your company name is censored though.
–
TarunMay 11 '11 at 5:54

Yeah sorry 'bout that. I had to, because I just realized that I signed a stupid legal agreement.... Thanks for the +1. :)
–
HannibalMay 11 '11 at 6:09

2

I think the only reasons to mention company names are (a) the company name is recognizable (b) the info is positive (c) the use of the company name adds credibility, e.g., "This is how Fog Creek does things" or "Google does that" or "Microsoft and Amazon do this differently, but both approaches seem to work OK". I mention my previous experiences at MS by name when appropriate and positive, but not my current (small and unrecognized) employer.
–
Ethel EvansMay 11 '11 at 18:27

The truth of it is every install of an operating system is different. Your goal when testing is to make sure your environment is as controlled as possible. This is both easier and harder with virtual machines.

The advantages are that you can create a brand new machine from an image in a matter of seconds. This lets you have the application get a fresh install (so no corruption from previous tests.) Control over your environment is paramount.

The major disadvantage is the shared processor/memory state. This is where your control over the environment starts to fall apart. What if someone else is doing something intensive and you don't have the resources it says you do?

At my last job we used VMs for our automated nightly tests and it worked out great. We knew they had the resources, and a fresh environment. We also had VMs of the various systems we supported. It worked like a charm.

Welcome to the era of the cloud.

For the record, we used VMWare. I'm not endorsing it or anything. It was easy to use, but I don't have a standard of comparison, nor did I do too much of the heavy lifting (although I did write Groovy integrations for starting and stopping VMs and that was pretty easy.)