In message from Joe Landman <landman at scalableinformatics.com> (Wed, 27
Apr 2005 15:51:49 -0400):
>Hi Art:
>> Any particular codes you have in mind? I used to play around with
>lots of DFT (LDA) codes. Back then, large systems were 256 x 256,
>with periodic BC's.
Most (practically all) DFT codes are not limited by eigenvalues
problem. The limiting stage is computation of 2-electron integrals and
fockian.
Yours
Mikhail Kuzminsky
Zelinsky Institute of Organic Chemistry
Moscow
> We used a number of eigensolvers, and eventually
>settled on LAPACK's zheev. Modeling supercells of much larger than
>64 atoms with 4 electronic basis states was a challenge using that
>code.
>> Do you have a particular model system in mind as well? A nice
>GAMESS
>model (or similar) might work out nicely. I would like to include
>some electronic structure codes in our (evolving) BBS system.
>>Joe
>>Art Edwards wrote:
>> This subject is pretty important to us. We run codes where the
>> bottleneck is eigensolving for matrices with a few thousand
>>elements.
>> Parallel eigen solvers are not impressive at this scale. In the dark
>> past, I did a benchmark on a Cray Y-MP using a vector eigen solve
>>and
>> got over 100x speedup. What I don't know is how this would compare
>>to
>> current compilers and CPU's. However the vector pipes are not very
>>deep
>> on any of the current processors except, possibly the PPC. So, I
>>would
>> like to see benchmarks of electronic structure codes that are bound
>>by
>> eigensolvers on a "true vector" machine.
>>>> Art Edwards
>>>> On Wed, Apr 27, 2005 at 01:15:42PM -0400, Robert G. Brown wrote:
>>>>>On Wed, 27 Apr 2005, Ben Mayer wrote:
>>>>>>>>>>>However, most code doesn't vectorize too well (even, as you say, with
>>>>>directives), so people would end up getting 25 MFLOPs out of 300
>>>>>MFLOPs
>>>>>possible -- faster than a desktop, sure, but using a multimillion
>>>>>dollar
>>>>>machine to get a factor of MAYBE 10 in speedup compared to (at the
>>>>>time)
>>>>>$5-10K machines.
>>>>>>>>What the people who run these centers have told me that a
>>>>supercomputer is worth the cost if you can get a speed up of 30x over
>>>>serial. What do others think of this?
>>>>>>I personally think that there is no global answer to this question.
>>>There is only cost-benefit analysis. It is trivially simple to
>>>reduce
>>>this assertion (by the people who run the centers, who are not
>>>exactly
>>>unbiased here:-) to absurdity for many, many cases. In either
>>>direction
>>>-- for some it might be worth it for a factor of 2 in speedup, for
>>>others it might NEVER be worth it at ANY speedup.
>>>>>>For example, nearly all common and commercial software isn't worth it
>>>at
>>>any cost. If your word processor ran 30x faster, could you tell?
>>> Would
>>>you care? Would it be "worth" the considerable expense of rewriting
>>>it
>>>for a supercomputer architecture to get a speedup that you could
>>>never
>>>notice (presuming that one could actually speed it up)?
>>>Sure it's an obvious exception, but the problem with global answers
>>>is
>>>they brook no exceptions even when there are obvious ones. If you
>>>don't
>>>like word processor, pick a suitable rendered computer game (zero
>>>productive value, but all sorts of speedup opportunities). Pick any
>>>software with no particular VALUE in the return or with a low
>>>OPPORTUNITY COST of the runtime required to run it.
>>>>>>A large number of HP computations are in the latter category. If I
>>>want
>>>to run a simple simulation that takes eight hours on a serial machine
>>>and that I plan to run a single time, is it worth it for me to spend
>>>a
>>>month recoding it to run in parallel in five minutes? Obviously not.
>>>If you argue that I should include the porting time in the
>>>computation
>>>of "speedup" then I'd argue that if I have a program that takes two
>>>years to run without porting and that takes a six months to port into
>>>a
>>>form that runs on a supercomputer in six months more, well, a year of
>>>MY
>>>life is worth it, depending on the actual COST of the "supercomputer"
>>>time compared to the serial computer time. Even in raw dollars, my
>>>salary for the extra year is nontrivial compared to the cost of
>>>purchasing and installing a brand-new cluster just to speed up the
>>>computation by a measley factor of two or four, depending on how you
>>>count.
>>>>>>So pay no attention to your supercomputer people's pronouncement.
>>> That
>>>number (or any other) is pulled out of, uh, their nether regions and
>>>is
>>>unjustifiable. Instead, do the cost-benefit analysis, problem by
>>>problem, using the best possible estimates you can come up with for
>>>the
>>>actual costs and benefits.
>>>>>>That very few people EVER actually DO this does not mean that it
>>>isn't
>>>the way it should be done;-)
>>>>>>>>>>:) I needed to do some CHARMM runs this summer. The X1 did not like
>>>>it
>>>>much (neither did I, but when the code is making references to punch
>>>>cards and you are trying to run it on a vector super, I think most
>>>>would feel that way), I ended up running it in parallel by a similar
>>>>method as yours. Worked great!
>>>>>>The easy way into cluster (or nowadays, "grid") computing, for sure.
>>> If
>>>your task is or can be run embarrassingly parallel, well, parallel
>>>scaling doesn't generally get much better than a straight line of
>>>slope
>>>one barring the VERY few problems that exhibit superlinear scaling
>>>for
>>>some regime....;-)
>>>>>>>>>>>If it IS a vector (or nontrivial parallel, or both) task, then the
>>>>>problem almost by definition will EITHER require extensive "computer
>>>>>science" level study -- work done with Ian Foster's book, Amalsi and
>>>>>Gottlieb for parallel and I don't know what for vector as it isn't my
>>>>>area of need or expertise and Amazon isn't terribly helpful (most
>>>>>books
>>>>>on vector processing deal with obsolete systems or are out of print,
>>>>>it
>>>>>seems).
>>>>>>>>So what we should really be trying to do is matching code to the
>>>>machine. One of the problems that I have run into is that unless one
>>>>is at a large center there are only one or two machines that provide
>>>>computing power. Where I am from we have a X1 and T3E. Not a very
>>>>good
>>>>choice between the two. There should be a cluster coming up soon,
>>>>which will give us the options that we need. ie Vector or Cluster.
>>>>>>No, what you SHOULD be doing is matching YOUR code to the cluster you
>>>design and build just for that code. With any luck, the cluster
>>>design
>>>will be a generic and inexpensive one that can be reused (possibly
>>>with
>>>minor reconfigurations) for a wide range of parallel problems. If
>>>your
>>>problem DOES trivially parallelize, nearly any grid/cluster of OTS
>>>computers capable of holding it in memory on (even) sneakernet will
>>>give
>>>you linear speedup.
>>>Given Cluster World's Really Cheap Cluster as an example, you could
>>>conceivably end up with a cluster design containing nodes that cost
>>>between $250 and $1000 each, including switches and network and
>>>shelving
>>>and everything, that can yield linear speedup on your code. Then you
>>>do
>>>your cost-benefit analysis, trade off your time, the value of the
>>>computation, the value of owning your own hardware and being able to
>>>run
>>>on it 24x7 without competition, the value of being able to redirect
>>>your
>>>hardware into other tasks when your main task is idle, any additional
>>>costs (power and AC, maybe some systems administration, maintenance).
>>>This will usually tell you fairly accurately both whether you should
>>>build your own local cluster vs run on a single desktop workstation
>>>vs
>>>run on a supercomputer at some center and will even tell you how many
>>>nodes you can/should buy and in what configuration to get the
>>>greatest
>>>net benefit.
>>>>>>Note that this process is still correct for people who have code that
>>>WON'T run efficiently on really cheap node or network hardware; they
>>>just have to work harder. Either way, the most important work is
>>>prototyping and benchmarking. Know your hardware (possibilities) and
>>>know your application. Match up the two, paying attention to how
>>>much
>>>everything costs and using real world numbers everywhere you can.
>>> AVOID
>>>vendor provided numbers, and look upon published benchmark numbers
>>>for
>>>specific micro or macro benchmarks with deep suspicion unless you
>>>really
>>>understand the benchmark and trust the source. For example, you can
>>>trust anything >>I<< tell you, of course...;-)
>>>>>>>>>>The manual for the X1 provides some information and examples. Are the
>>>>Apple G{3,4,5} the only processors who have real vector units? I have
>>>>not really looked at SSE(2), but remember that they were not full
>>>>precision.
>>>>>>What's a "real vector unit"? On chip? Off chip? Add-on board?
>>>Integrated with the memory and general purpose CPU (and hence
>>>bottlenecked) how?
>>>>>>Nearly all CPUs have some degree of vectorization and parallelization
>>>available on chip these days; they just tend to hide a lot of it from
>>>you. Compilers work hard to get that benefit out for you in general
>>>purpose code, where you don't need to worry about whether or not the
>>>unit is "real", only about how long it takes the system to do a
>>>stream
>>>triad on a vector 10 MB long. Code portability is a "benefit" vs
>>>code
>>>specialization is a "cost" when you work out the cost-benefit of
>>>making
>>>things run on a "real vector unit". I'd worry more about the times
>>>returned by e.g. stream with nothing fancy done to tune it than how
>>>"real" the underlying vector architecture is.
>>>>>>Also, if your problem DOES trivially parallelize, remember that you
>>>have
>>>to compare the costs and benefits of complete solutions, in place.
>>> You
>>>really have to benchmark the computation, fully optimized for the
>>>architecture, on each possible architecture (including systems with
>>>"just" SSE but perhaps with 64 bit memory architectures and ATLAS for
>>>linear algebra that end up still being competitive) and then compare
>>>the
>>>COST of those systems to see which one ends up being cheaper.
>>> Remember
>>>that bleeding edge systems often charge you a factor of two or more
>>>in
>>>cost for a stinkin' 20% more performance, so that you're better off
>>>buying two cheap systems rather than one really expensive one IF your
>>>problem will scale linearly with number of nodes.
>>>>>>I personally really like the opteron, and would commend it to people
>>>looking for a very good general purpose floating point engine. I
>>>would
>>>mistrust vendor benchmarks that claim extreme speedups on vector
>>>operations for any code big running out of memory unless the MEMORY
>>>is
>>>somehow really special. A Ferrari runs as fast as a Geo on a crowded
>>>city street.
>>>>>>As always, your best benchmark is your own application, in all its
>>>dirty
>>>and possibly inefficiently coded state. The vendor specs may show 30
>>>GFLOPS (for just the right code running out of L1 cache or out of
>>>on-chip registers) but when you hook that chip up to main memory with
>>>a
>>>40 ns latency and some fixed bandwidth, it may slow right down to
>>>bandwidth limited rates indistinguishable from those of a much slower
>>>chip.
>>>>>>>>>>>For me, I just revel in the Computer Age. A decade ago, people were
>>>>>predicting all sorts of problems breaking the GHz barrier. Today
>>>>>CPUs
>>>>>are routinely clocked at 3+ GHz, reaching for 4 and beyond. A decade
>>>>>>>>I just picked up a Semptron 3000+, 1.5GB RAM, 120GB HD, CD-ROM,
>>>>video,
>>>>10/100 + intel 1000 Pro for $540 shipped. I was amazed.
>>>>>>The Opterons tend to go for about twice that per CPU, but they are
>>>FAST,
>>>especially for their actual clock. The AMD-64's can be picked up for
>>>about the same and they too are fast. I haven't really done a
>>>complete
>>>benchmark run on the one I own so far, but they look intermediate
>>>between Opteron and everything else, at a much lower price.
>>>>>> rgb
>>>>>>--
>>>Robert G. Brown http://www.phy.duke.edu/~rgb/>>>Duke University Dept. of Physics, Box 90305
>>>Durham, N.C. 27708-0305
>>>Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu>>>>>>>>>_______________________________________________
>>>Beowulf mailing list, Beowulf at beowulf.org>>>To change your subscription (digest mode or unsubscribe) visit
>>>http://www.beowulf.org/mailman/listinfo/beowulf>>>>>>--
>Joseph Landman, Ph.D
>Founder and CEO
>Scalable Informatics LLC,
>email: landman at scalableinformatics.com>web : http://www.scalableinformatics.com>phone: +1 734 786 8423
>fax : +1 734 786 8452
>cell : +1 734 612 4615
>>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org>To change your subscription (digest mode or unsubscribe) visit
>http://www.beowulf.org/mailman/listinfo/beowulf