Microsoft HPC set to take over the world, allegedly

Comment If our English readers need an ego boost, Microsoft has just chosen the UK for the worldwide launch of Windows Compute Cluster Server 2003, its HPC (High Performance Computing) offering.

This is because high performance and technical computing is a growth market in the UK and a lot of the basic work on it was done at places like Professor Simon Cox's Microsoft HPC Institute at Southampton University.

Of course, if I go to Sun HPC presentations, they have their own academic HPC sites, complete with academics speaking about Sun products (most computer companies have academic programs). There's indeed a growing demand for HPC - although, as I watched each slide in Microsoft's Live Meeting launch presentation arrive two to three minutes after the speaker started talking about it, I was reminded that a lot of applications that are too slow aren't actually compute bound.

What Microsoft is doing is commoditising HPC and making it easier to use. Its offering looks to be well crafted and appears to give new users an immediate performance boost for an affordable number of euros, as is necessary if you want happy customers.

However, these customers probably won't get the full benefits, of course, unless their algorithms suit HPC-style parallel processing. Nevertheless, Microsoft has been targeting ISVs already, so some standard applications will provide some of the benefits of HPC off the shelf. In fact, one of the most frightening prospects held out at an HPC workshop held recently was of Excel running as an HPC application on the server - now you really can run the enterprise on a spreadsheet. No, I don't think I was having nightmares.

In fact, I suspect a lot of customers might well get a similar performance boost to that from HPC simply by buying one of the new, cheap, fast, multicore boxes. But this doesn't matter because HPC is very cost effective and lots of Microsoft customers will get a buzz from running on HPC even if they don't really need to. So Microsoft will probably pick up a lot of new customers, who'll help finance its capture of the high-end HPC market in a year or so. And, to be fair, Compute Cluster Server looks reasonably well engineered - it'll probably catalyse price reductions all round (cf. SQLServer).

But at the high end, there is strong competition for this Microsoft HPC offering - with a proven record of accomplishment. According to Professor Cox, Microsoft's solution targets tightly-coupled compute-bound applications, of which there are enough (as opposed to, say, grid computing, which tackles more loosely-coupled applications); but its real unique selling point over existing HPC technology seems to be its ease-of-use - it is firmly integrated with Microsoft's mainstream Visual Studio environment - and its low cost. Most existing, and effective, HPC solutions need expensive specialists.

Professor Ken Bell, Pro-Vice-Chancellor Queen's University Belfast, confirms this: "We are seeking to expand the use of high performance computing to user groups who do not have the in-depth UNIX or Linux skills usually associated with high performance computing. These users have requirements that go beyond the capability of their desktop systems to run financial modelling, imaging and engineering applications. Windows Compute Cluster Server offers Queens University users the ability to solve complex computational problems in a user friendly environment."

However, if you do really need HPC, I'm still somewhat concerned about the possible impact on developers - currently, people who can write technical applications for HPC clusters are a bit of a special breed and I suspect that there are many ways for developers to write applications that can't take full advantage of them.

For a start, I suspect hoary old Amdahl's Law still applies. Perhaps this is all looked after by the Windows Compute Cluster Server scheduler, but once scheduled, what happens to a unit of work that doesn't behave as the scheduler expected? In the worst case, it probably times out (as long as the developer remembered to code time-out) but enough time-outs can really affect the overall throughput of work.

I'll be interested to see how Microsoft disseminates HPC "good practice" for developers - I've already seen trainers licking their lips over the opportunities this represents, and what is good for trainers may not be so good for developer budgets.

Anyway, I'm not going to do any more of Microsoft's marketing for this HPC offering. I think that HPC is something that developers at least need to be aware of; and that Microsft is strongly committed to competing in this area. You can download an evaluation of Compute Cluster Server here and you'll be able to buy it in early August if you like what you get.