The current virtualisation splits one big physical server into smaller virtual servers. But is it possible to combine several physical servers into 1 virtual server ?

UPDATE: We have several different systems which require their own server to perform CPU intensive jobs. What we are trying to do is to create 2 virtual servers spread across 6-8 smaller capacity servers. The virtualization software we investigated takes 1 big physical server and slices into 2-3 smaller capacity servers.

We are also looking into dynamically increasing CPU of virtual servers, so that we could use full CPU capacity when one of them is using less capacity.

6 Answers
6

What you're describing is cluster computing (i.e. Google "Beowulf Cluster"). It doesn't behave quite the way you're describing (i.e. it won't pretend to be one OS, exactly, and you'll need some kind of job scheduler/dispatcher to tell which machines to run which processes unless you've specifically written your code to use the MPICH libraries are something similar to automatically distribute across all machines).

The reason that it doesn't work like a simple "this is just one OS" interface is because you don't want to conceal (and thus inhibit tuning around) the gigantic latencies (in terms of processing cycles) involved in having bits of memory retrieved across a network.

For code that is single-threaded, but needs to be dispatched from a central location, there are small tools such as "dsh" designed to allow you to parallelize jobs easily across multiple machines via SSH.

Without more specifics about what you're trying to do, it's difficult to offer more precise advice.

If it is technically possible to consolidate the functionality of the 10 physical machines into one physical machine (and it doesn't break something important like security/compliance requirements) then you can certainly do this in a virtual environment as well.

Hope this answer can help you (even if several months late). You idea is not utopian, there are solutions for your problem. As voretaq7 user said, virtualization usually means hardware consolidation, bout you want to do hardware aggregation.

The solutions I know (please note they are only for Linux) are: LinuxPMI/OpenMosix, OpenSSI, Kerrighed and ScaleMP. All open-source except ScaleMP. I don't have experience using any of them.

You can't do this transparently - I've never seen any transparent distributed shared memory system on Linux that would do this in a completely background manner. All Unix or Linux based DSM systems that I've ever heard of require the program to be linked into a library.

If your application can be clustered then you could just run the servers as cluster nodes, otherwise you may have to buy a bigger server. Fortunately this is not all that expensive these days and is probably much cheaper than attempting to retrofit parallelism to an existing application (if this is even possible with your apps).