From ajt at rri.sari.ac.uk Wed Oct 1 06:49:11 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Wed, 01 Oct 2008 11:49:11 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
<05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr>
Message-ID: <48E355A7.6070008@rri.sari.ac.uk>
J?rgen Kn?dlseder wrote:
> Hi Tony,
>
> I'm in the same situation as your are: I'm running an openMosix cluster,
> but since it's more and more
> difficult to integrate new hardware with an old 2.4.26 kernel I think
> that I have to move to MOSIX2.
> I just got the latest version of MOSIX2 sent from Amnon Barak (I'm using
> the cluster for academic
> work), and started to play around with installations ... yet I had some
> kernel crash problems that
> I could not yet resolve on one of the machines. So I guess without the
> $1,000 per year support
> fee it'll also be difficult to run a MOSIX2 cluster ...
Hello, J?rgen.
I ported 2.4.26-om1 to compile it using the 'new' GNU tool-chain and run
it under Ubuntu 6.06.1. I also patched the kernel.org 2.4.32 sources for
openMosix (to get SATA drivers). However, I couldn't get openMosix
process migration working properly between 2.4.32-om1 kernels...
Ironically, process migration worked between a 2.4.26-om1 and 2.4.32.om1
kernel! Anyway, I gave up the attempt at 2.4.32-om1, and just continued
to use my version of 2.4.26-om1 in production with SCSI controllers. If
you're interested, my Ubuntu 6.06.1 openMosix deb's are at:
http://bioinformatics.rri.sari.ac.uk/openmosix
Are you going to continue using MOSIX2?
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From ajt at rri.sari.ac.uk Wed Oct 1 07:10:13 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Wed, 01 Oct 2008 12:10:13 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To:
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
Message-ID: <48E35A95.7020908@rri.sari.ac.uk>
Vincent Diepeveen wrote:
> I agree tony that paying for such crap is not very good idea.
Hello, Vincent.
I don't think MOSIX2 is crap!
However, I don't like the idea of having to pay for 'updates'.
> You might want to move to open-ssi in this case; the project is alive
> and there is in theory work getting
> performed on support for cards over infiniband as well.
I have looked at OpenSSI, which uses the openMosix load-balancer, but
process migration is more coarsely grained than in openMosix. Only the
active pages of the user context of openMosix processes are migrated.
I've been looking at alternatives and I think Kerrighed looks very
promising but, in our hands, Kerrighed is very fragile: I've mentioned
on this list before that if one Kerrighed node goes down you lose the
entire cluster. We've been talking to Christine Morin's group at INRIA
and they tell us that the next release of Kerrighed with be more robust:
http://www.kerrighed.org
> Most importantly is that you are gonna get more replies.
Yes, thanks for yours :-)
> Additionally the manner open-ssi implements shared memory is very
> transparant; in principle on each write it migrates a page
> to the node writing.
>
> Maybe the only big lack of open-ssi is its limited support so far for
> highend network cards.
What bothers me about OpenSSI is that it's based on an open-sourced
version HP's (now Compaq) discontinued commercial product "non-stop
clusters for Unix". The OpenSSI project also came in for a lot of
criticism from the openMosix community for stealing ideas, so my
concerns about it might not be all that well founded ;-)
The main reason I didn't use OpenSSI, previously, was that many features
had not been implemented fully and, like Kerrighed, it wasn't really a
viable option for a 'production' cluster even though it was interesting
as a research project. What makes me take MOSIX2 seriously now is that
it is a commercially supported 'product' with all the same virtues (and
most of the vices) of openMosix.
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From ajt at rri.sari.ac.uk Wed Oct 1 07:13:00 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Wed, 01 Oct 2008 12:13:00 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <200810010212.00791.mm@yuhu.biz>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk>
<200810010212.00791.mm@yuhu.biz>
Message-ID: <48E35B3C.20401@rri.sari.ac.uk>
Marian Marinov wrote:
> There are a few developers that continue to work on openMosix and to port it
> to 2.6 kernels.
>
> They forked a project called LinuxPMI - Linux Process Migration Infrastructure
>
> http://linuxpmi.org
>
> Currently the site is unavailable but there was one Russion guy who commented
> big parts of the openMosix code he started workin on that in the begining of
> February this year and in May he and a few other developers forked LinuxPMI
> from openMosix.
>
> I had a working 2.6 kernel with openMosix functionality in Jul but I never had
> the chance to test its process migration capabilities as the nodes from my
> home cluster died :(
Hello, Marian.
Great! I didn't know about LinuxMPI: I've not been following openMosix
developments recently. I can't access their web site - Is it mirrored
anywhere?
Thanks,
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From ajt at rri.sari.ac.uk Wed Oct 1 07:33:38 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Wed, 01 Oct 2008 12:33:38 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <48E35B3C.20401@rri.sari.ac.uk>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk>
<200810010212.00791.mm@yuhu.biz> <48E35B3C.20401@rri.sari.ac.uk>
Message-ID: <48E36012.1070406@rri.sari.ac.uk>
Tony Travis wrote:
> [...]
> Hello, Marian.
>
> Great! I didn't know about LinuxMPI: I've not been following openMosix
> developments recently. I can't access their web site - Is it mirrored
> anywhere?
Following up my own message, I meant LinuxPMI of course ;-)
Not much evidence that the LinuxPMI project is still active, though.
Google reports a lot of hits for the pending DNS de-registration of
"linuxmpi.com'...
Anyone else know what's happening with LinuxPMI?
Thanks,
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From ajt at rri.sari.ac.uk Wed Oct 1 08:16:29 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Wed, 01 Oct 2008 13:16:29 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To:
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
<05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr>
<48E355A7.6070008@rri.sari.ac.uk>
Message-ID: <48E36A1D.2020901@rri.sari.ac.uk>
J?rgen Kn?dlseder wrote:
> Hi Tony,
>
> in fact, I also patched SATA drivers in the 2.4.26-om kernel. Yet I
> could so far not manage to
> get the kernel working for my latest DELL PE1950 with a Perc 6i driver
> ... maybe the megaraid_sas
> driver I use is outdated ...
Hello, Jurgen.
I did have 'some' SATA support using 2.4.27-om1 from the ClusterKnoppix
sources, but the drivers were very old. The 2.4 kernel *is* still
supported at Kernel.org and the more recent SATA drivers worked a lot
better. However, I don't want to invest a lot of time and effort keeping
2.4.xx-om1 alive without a critical mass of other openMosix users and
developers. Sadly, whatever the virtues of openMosix, I think it is now
becoming unsupportable for use on a 'production' Beowulf cluster.
> I also search for LinuxPMI project information, but without any success.
> So I share your feeling that
> this projects is probably dead ...
It's probably just as well, or I would be tempted to join in ;-)
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 1 08:52:29 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Wed, 1 Oct 2008 14:52:29 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
Message-ID:
On Tue, 30 Sep 2008, Donald Becker wrote:
> Ahhh, your first flawed assumption.
>
> You believe that the OS needs to be statically provisioned to the nodes.
> That is incorrect.
Well, you also make the flawed assumption that the best technical
solutions are always preferred. From my position I have seen many
cases where political or administrative reasons have very much
restricted the choice of technical solutions that could be used. Other
reasons are related to the lack of flexibility from ISVs which provide
applications in binary form only and make certain assumptions about
the way the target cluster works. Yet another reason is the fact that
a solution like Scyld's limits the whole cluster to running one
distribution (please correct me if I'm wrong), while a solution with
node "images" allows mixing Linux distributions at will.
> The only times that it is asked to do something new (boot, accept a
> new process) it's communicating with a fully installed, up-to-date
> master node. It has, at least temporarily, complete access to a
> reference install.
I think that this is another assumption that holds true for the Scyld
system, but there are situations where this is not true. Some years
ago I have developed a rudimentary batch system for which the master
node only contacted the first node allocated/desired for the job; this
node was then responsible to contact the other nodes allocated/desired
and start the rest of the job. This was very much modelled after the
way the naive rsh/ssh based launchers for MPI jobs work: once mpirun
is running, there is no connection to the master node, only between
the node where mpirun is running and the rest of the nodes specified
in the hosts file. I think that Torque also has a similar design
(Mother Superior being in control of the job), but I haven't look
closely at the details so I might be wrong.
> If you design a cluster system that installs on a local disk, it's
> very difficult to adapt it to diskless blades. If you design a
> system that is as efficient without disks, it's trivial to
> optionally mount disks for caching, temporary files or application
> I/O.
If you design a system that is flexible enough to allow you to use
either diskless or diskfull installs, what do you have to loose ?
The same node "image" can be used in several ways:
- copied to the local disk and booted from there (where the copying
could be done as a separate operation followed by a reboot or it can
be done from initrd)
- used over NFS-root
- used as a ramdisk, provided that the node "image" is small enough
Note: I have used "image" in this and previous e-mails to signify the
collection of files that the node needs for booting; most likely this
is not a FS image (like an ISO one), but it could also be one. Various
documents call this a "virtual node FS", "chroot-ed FS", etc.
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rgb at phy.duke.edu Wed Oct 1 10:18:20 2008
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Wed, 1 Oct 2008 10:18:20 -0400 (EDT)
Subject: [Beowulf] precise synchronization of system clocks
In-Reply-To: <20081001020421.GA8126@compegg.wr.niftyegg.com>
References:
<20081001020421.GA8126@compegg.wr.niftyegg.com>
Message-ID:
On Tue, 30 Sep 2008, Nifty niftyompi Mitch wrote:
> Also while focusing on network/transport in this discussion none of us
> made a comment on rotational latency as a source of uncertainty for the
> kernel state. If we had the ability to synchronize the systems exactly
> starting a process would lag for want of rotational/seek disk latency
> in beowulf'y.
Sure, but starting processes is presumed to occur once and then (an
efficient computation) proceeds for a long time. All the parallelized
tasks effectively arrive at the same point at the computation at the
first barrier AFTER all this is finished. If the communications involve
giving the kernel a slice at (say) the end when the barrier is released
for all nodes at "the same time", then the distributed kernels CAN have
roughly the same state IF one suppresses enough of the "random" sources
of state-noise -- asynchronous demands for the kernel's attention. To
the extent that the kernel can accumulate all of its regular
housekeeping and do it an elective time, if one gets the system clocks
together within a usec and the kernel does its elective work on clock
ticks, that work will end up being done (mostly) synchronously across
the nodes.
Truthfully, all one is trying to do is to generalize your parallel
process to have a double synchronous barrier, with one phase of the
computation being "kernel housekeeping".
Compute (in parallel) -> barrier (IPCs) -> Kernel (in parallel) ->
barrier -> Compute -> barrier -> Kernel -> barrier ... ad inifinitum.
If the work done by the kernel is fairly tightly bounded -- predictably
completes in (say) 100 usec (which is a LOT of time, far more than one
almost ever sees one STILL has 900 usec per tick to work on your compute
task. If the kernel (more reasonably) completes in 1-10 usec your
cluster should have a 99+% duty cycle but avoid the "noise" that
desynchronizes everything.
>
> Shared memory machines and transports will behave differently.
>
> The very high accuracy and high precision clock synchronization is a very real
> problem for some data gathering systems. Once the data is gathered the
> computation should be less sensitive. These are different problems and
> might be addressed by the data sampling devices.
>
> Synchronization brings problems.... for example a well synchronized campus
> can hammer yp server and file servers when cron triggers the same actions on
> 5000+ systems... I try never to fetchmail at the hour, half hour...
>
> I suspect that some system cron tasks should no longer run from cron. Common
> housekeeping tasks necessary for system health should be run via the batch system
> in a way that is fashionably late enough to not hammer site services.
Absolutely. In fact, you'd want the nodes to be isolated and not
running ANY of this stuff, I'd guess. You'd want the nodes to have
quiescent, non-demanding hardware (except for devices doing the bidding
of the running parallel process) so that nothing "random" needed to be
done that couldn't be saved for the kernel slices.
> One site service of interest is AC power. A modern processor sitting
> in an idle state that then starts a well optimized loop will jump from
> a couple of watts to 100 watts in as many clocks as the set of pipelines
> is deep behind the instruction decode and instruction cache fill. A 1000
> processor (4000 cores) might jump from 4000 watts to 100000 watts in the
> blink of an eye (err did the lights blink). Buffer that dI/dT through
> the PS and it is less but still interesting on the mains which are synchronized.
Interesting. I never have seen the lights blink although I don't run
synchronous computations. One wonders if the power supply capacitors
(which should be quite large, I would think) don't soak up the
transient, though, even on very large clusters. Also, I think that the
power differential is smaller than you are allowing for -- I don't think
most idle processors draw "no" power...
rgb
>
>
>
>
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From prentice at ias.edu Wed Oct 1 11:30:10 2008
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 01 Oct 2008 11:30:10 -0400
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
Message-ID: <48E39782.4020006@ias.edu>
In the ongoing saga that is my new cluster, we were just told today that
Cisco is no longer manufacturing DDR IB cables, which we, uhh, need.
Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
If so, why would anyone spec a brand new cluster with DDR?
--
Prentice
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From prentice at ias.edu Wed Oct 1 11:44:01 2008
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 01 Oct 2008 11:44:01 -0400
Subject: [Beowulf] Advanced Clustering's Breakin
References: 20670.89.180.225.196.1217678257.squirrel@www.di.fct.unl.pt
Message-ID: <48E39AC1.7080001@ias.edu>
> We have a tool on our website called "breakin" that is Linux 2.6.25.9
> patched with K8 and K10f Opteron EDAC reporting facilities. It can
> usually find and identify failed RAM in fifteen minutes (two hours at
> most). The EDAC patches to the kernel aren't that great about naming
> the correct memory rank, though.
>
> Make sure you have multibit (sometimes says 4-bit) ECC enabled in your BIOS.
>
> http://www.advancedclustering.com/software/breakin.html
I've been using breakin for the past week or two on my new cluster. I
get some results that seem to be inconsistent. For example on a node
I'll get this:
Test | Pass | Fail | Last Message
------------------------------------------
hdhealth | 315 | 0 | No disk devices found
Then in the log section:
00h 57m 40s: Disabling burnin test 'hdhealth'
If I reboot and restart the testing, it will see a hard disk. Why is
breaking not always seeing the disk?
I've tried to dump logs to a USB drive, but breakin refuses to mount the
correct partition on my usb drive (/dev/sdb vs. /dev/sdb1, or vice versa).
I sent e-mail to Advanced Clustering regarding these issues, but didn't
get any response, so I"m hoping I have better luck here.
--
Prentice
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From jurgen.knodlseder at cesr.fr Wed Oct 1 07:54:44 2008
From: jurgen.knodlseder at cesr.fr (=?ISO-8859-1?Q?J=FCrgen_Kn=F6dlseder?=)
Date: Wed, 1 Oct 2008 13:54:44 +0200
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <48E355A7.6070008@rri.sari.ac.uk>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
<05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr>
<48E355A7.6070008@rri.sari.ac.uk>
Message-ID:
Hi Tony,
in fact, I also patched SATA drivers in the 2.4.26-om kernel. Yet I
could so far not manage to
get the kernel working for my latest DELL PE1950 with a Perc 6i
driver ... maybe the megaraid_sas
driver I use is outdated ...
I also search for LinuxPMI project information, but without any
success. So I share your feeling that
this projects is probably dead ...
J?rgen
Le 1 oct. 08 ? 12:49, Tony Travis a ?crit :
> J?rgen Kn?dlseder wrote:
>> Hi Tony,
>> I'm in the same situation as your are: I'm running an openMosix
>> cluster, but since it's more and more
>> difficult to integrate new hardware with an old 2.4.26 kernel I
>> think that I have to move to MOSIX2.
>> I just got the latest version of MOSIX2 sent from Amnon Barak (I'm
>> using the cluster for academic
>> work), and started to play around with installations ... yet I had
>> some kernel crash problems that
>> I could not yet resolve on one of the machines. So I guess without
>> the $1,000 per year support
>> fee it'll also be difficult to run a MOSIX2 cluster ...
>
> Hello, J?rgen.
>
> I ported 2.4.26-om1 to compile it using the 'new' GNU tool-chain
> and run it under Ubuntu 6.06.1. I also patched the kernel.org
> 2.4.32 sources for openMosix (to get SATA drivers). However, I
> couldn't get openMosix process migration working properly between
> 2.4.32-om1 kernels...
>
> Ironically, process migration worked between a 2.4.26-om1 and
> 2.4.32.om1 kernel! Anyway, I gave up the attempt at 2.4.32-om1, and
> just continued to use my version of 2.4.26-om1 in production with
> SCSI controllers. If you're interested, my Ubuntu 6.06.1 openMosix
> deb's are at:
>
> http://bioinformatics.rri.sari.ac.uk/openmosix
>
> Are you going to continue using MOSIX2?
>
> Tony.
> --
> Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
> and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
> tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
> mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From mm at yuhu.biz Wed Oct 1 09:03:10 2008
From: mm at yuhu.biz (Marian Marinov)
Date: Wed, 1 Oct 2008 16:03:10 +0300
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <48E36012.1070406@rri.sari.ac.uk>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E35B3C.20401@rri.sari.ac.uk> <48E36012.1070406@rri.sari.ac.uk>
Message-ID: <200810011603.10435.mm@yuhu.biz>
I lost touch with Juri (the guy who started the documentation project of
openMosix) 2-3 months ago.
More info about LinuxPMI you can find on Freenode #linuxpmi or #openmosix
As far as I can see the domain is registered until next year and there is some
routing problem New York before the server.
I'm sorry but I do not know of any mirrors as Juri's machine was the central
storage of the documented patches.
Hi actually split the openMosix into series of patches so that it can be
ported to 2.6 easier.
Regards
Marian
On Wednesday 01 October 2008 14:33:38 Tony Travis wrote:
> Tony Travis wrote:
> > [...]
> > Hello, Marian.
> >
> > Great! I didn't know about LinuxMPI: I've not been following openMosix
> > developments recently. I can't access their web site - Is it mirrored
> > anywhere?
>
> Following up my own message, I meant LinuxPMI of course ;-)
>
> Not much evidence that the LinuxPMI project is still active, though.
> Google reports a lot of hits for the pending DNS de-registration of
> "linuxmpi.com'...
>
> Anyone else know what's happening with LinuxPMI?
>
> Thanks,
>
> Tony.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From becker at scyld.com Wed Oct 1 12:35:37 2008
From: becker at scyld.com (Donald Becker)
Date: Wed, 1 Oct 2008 09:35:37 -0700 (PDT)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
Message-ID:
On Wed, 1 Oct 2008, Bogdan Costescu wrote:
> On Tue, 30 Sep 2008, Donald Becker wrote:
> > Ahhh, your first flawed assumption.
> > You believe that the OS needs to be statically provisioned to the nodes.
> > That is incorrect.
> Well, you also make the flawed assumption that the best technical
> solutions are always preferred. From my position I have seen many
...
> a solution like Scyld's limits the whole cluster to running one
> distribution (please correct me if I'm wrong), while a solution with
> node "images" allows mixing Linux distributions at will.
That's correct. Our model is that a "cluster" is a single system -- and a
single install.
That's for a good reason: To keep the simplicity and consistency of
managing a single installation, you pretty much can have... only a single
installation.
There is quite a bit of flexibility. The system automatically detects the
hardware and loads the correct kernel modules. Nodes can be specialized,
including mounting different file systems and running different start-up
scripts. But the bottom line is that to make the assertion that remote
processes will run the same as local processes, they have to be running
pretty much the same system.
If you are running different distributions on nodes, you discard many of
the opportunities of running a cluster. More importantly, it's much
more knowledge- and labor-intensive to maintain the cluster while
guaranteeing consistency.
> > The only times that it is asked to do something new (boot, accept a
> > new process) it's communicating with a fully installed, up-to-date
> > master node. It has, at least temporarily, complete access to a
> > reference install.
>
> I think that this is another assumption that holds true for the Scyld
> system, but there are situations where this is not true.
Yes, there are scenarios where you want a different model. But "connected
during important events" is true for most clusters. We discard the
ability for a node to boot and run independently in order to get the
advantages of zero-install, zero-config consistent compute nodes.
> > If you design a cluster system that installs on a local disk, it's
> > very difficult to adapt it to diskless blades. If you design a
> > system that is as efficient without disks, it's trivial to
> > optionally mount disks for caching, temporary files or application
> > I/O.
>
> If you design a system that is flexible enough to allow you to use
> either diskless or diskfull installs, what do you have to loose ?
In theory that sounds good. But historically changing disk-based
installations to work on diskless machines has been very difficult, and
the results unsatisfactory. Disk-based installations want to do selective
installation based on the hardware present, and write/modify many links
and configuration files on installation -- many more than they "need" to.
> The same node "image" can be used in several ways:
> - copied to the local disk and booted from there (where the copying
> could be done as a separate operation followed by a reboot or it can
> be done from initrd)
> - used over NFS-root
> - used as a ramdisk, provided that the node "image" is small enough
While memory follows the price-down capacity-up curve, we aren't quite to
the point where holding a full OS distribution in memory is negligible.
Most distributions (all the commercially interesting ones) are
workstation-oriented, and the trade-off is "disk is under $1/GB, so we
will install everything". It's foreseeable that holding an 8GB install
image in memory will be trivial, but that will be a few years in the
future, not today. And we will need better VM and PTE management to make
it efficient.
--
Donald Becker becker at scyld.com
Penguin Computing / Scyld Software
www.penguincomputing.com www.scyld.com
Annapolis MD and San Francisco CA
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Wed Oct 1 13:23:00 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Wed, 01 Oct 2008 13:23:00 -0400
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E39782.4020006@ias.edu>
References: <48E39782.4020006@ias.edu>
Message-ID: <48E3B1F4.3000702@scalableinformatics.com>
Prentice Bisbal wrote:
> In the ongoing saga that is my new cluster, we were just told today that
> Cisco is no longer manufacturing DDR IB cables, which we, uhh, need.
>
> Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
>
> If so, why would anyone spec a brand new cluster with DDR?
Hmmm.... Cisco isn't the only provider of IB cables. Last I understood,
they buy theirs from others.
Cisco does appear to be transitioning to *other* technologies. Again,
they aren't the only IB provider out there (seems such a shame, gobbling
up TopSpin and then effectively discarding them).
Qlogic, Voltaire, Mellanox are happy to sell you stuff. You can even
call some of the tier2+ players and they will help.
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 1 13:39:37 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Wed, 1 Oct 2008 19:39:37 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <48E2C6C1.60108@neuralbs.com>
References:
<48E2C6C1.60108@neuralbs.com>
Message-ID:
On Tue, 30 Sep 2008, Eric Thibodeau wrote:
> This has given me much flexibility and a very fast path to upgrade
> the nodes (LIVE!) since they would only need to be rebooted if I
> changed the kernel. I can install/upgrade the node's environment by
> simply chrooting into it and using the node's package manager and
> utilities as if it were a regular system).
Only the first is an advantage of using NFS-root; the second is shared
by most methods that use a node "image". However random installations
or modifications of configuration file within the chroot become very
difficult to reproduce when you build the next node "image" - either
scripting everything or using cfengine/puppet/etc. can save a lot of
time in the long run, despite the initial effort to set up.
> But I am in a special case where, if I break the cluster, I can fix
> it quickly and I always have a backup copy of the boot "root" image
> ready to switch to if my fiddling goes wrong.
Why not keeping several "images" around and only point the nodes to
mount the one considered current or "good" ?
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From prentice at ias.edu Wed Oct 1 13:49:43 2008
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 01 Oct 2008 13:49:43 -0400
Subject: [Beowulf] Advanced Clustering's Breakin
In-Reply-To:
References: <48E39AC1.7080001@ias.edu>
Message-ID: <48E3B837.4020306@ias.edu>
billycrook at gmail.com wrote:
> On 2008-10-01, Prentice Bisbal wrote:
> ...
>> I sent e-mail to Advanced Clustering regarding these issues, but didn't
>> get any response, so I"m hoping I have better luck here.
>
> Prentice,
>
> I'm the primary support contact at Advanced Clustering. Our Support
> email address is Support at AdvancedClustering.com. I looked through our
> ticket system, but didn't see a message from you there. We test
> breakin against clusters we sell, but would like it to work on as much
> hardware as possible. When hdhealth doesn't see hard drives, is
> usually because of some raid hardware sheilding SMART data, or the
> hard drive controller driver not being in the kernel yet. When its
> intermittent, it could be cabeling, heat related, or the drives
> actually failing.
>
> What hardware specifically are you testing? The motherboard model,
> and any storage adapters involved would help.
>
> I will open a ticket momentarily from our helpdesk to prentice at ias.edu
> to look at this issue off list and where it is accessible to all of
> our support personnel until the issue is resolved.
>
> Thanks,
> Billy Crook
>
> Advanced Clustering Customer Support
> Support: 866.802.8222 x2
> Support: 913.643.0300 x2
> Fax: 913.378.9117
>
Billy,
I got your support e-mail off list, and replied already. We can continue
this disucssion off-list.
Thanks for the help. I sent my previous e-mail to some address on the
web page for breakin.
--
Prentice
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From prentice at ias.edu Wed Oct 1 13:59:05 2008
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 01 Oct 2008 13:59:05 -0400
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E3B1F4.3000702@scalableinformatics.com>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com>
Message-ID: <48E3BA69.2000405@ias.edu>
Joe Landman wrote:
> Prentice Bisbal wrote:
>> In the ongoing saga that is my new cluster, we were just told today that
>> Cisco is no longer manufacturing DDR IB cables, which we, uhh, need.
>>
>> Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
>>
>> If so, why would anyone spec a brand new cluster with DDR?
>
> Hmmm.... Cisco isn't the only provider of IB cables. Last I understood,
> they buy theirs from others.
I'm sure you're right. It's my understanding that IB is a
well-documented standard, so all IB cables should be created equally,
but you know vendors, they'll say that their cables are more equal than
others and refuse support if we're not using all Cisco kit.
And what is the status of DDR? Are people still using it, or has it
already been replaced by QDR in the marketplace?
>
> Cisco does appear to be transitioning to *other* technologies. Again,
> they aren't the only IB provider out there (seems such a shame, gobbling
> up TopSpin and then effectively discarding them).
What *other* technolgies are you talking about QDR IB, or something
other than IB altogether, like 10 Gb Ethernet, or something all new and
proprietary?
>
> Qlogic, Voltaire, Mellanox are happy to sell you stuff. You can even
> call some of the tier2+ players and they will help.
>
> Joe
>
>
--
Prentice
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From jan.heichler at gmx.net Wed Oct 1 14:19:11 2008
From: jan.heichler at gmx.net (Jan Heichler)
Date: Wed, 1 Oct 2008 20:19:11 +0200
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E3BA69.2000405@ias.edu>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com>
<48E3BA69.2000405@ias.edu>
Message-ID: <1848483940.20081001201911@gmx.net>
Hallo Prentice,
Mittwoch, 1. Oktober 2008, meintest Du:
PB> And what is the status of DDR? Are people still using it, or has it
PB> already been replaced by QDR in the marketplace?
DDR is still the most important Infiniband in the market. DDR provides enough bandwidth for most applications at the moment because most applications suffer from latency - not limited bandwidth. QDR is more interesting for building spine networks. And QDR might get more important if we see more cores in typical compute nodes.
>> Cisco does appear to be transitioning to *other* technologies. Again,
>> they aren't the only IB provider out there (seems such a shame, gobbling
>> up TopSpin and then effectively discarding them).
PB> What *other* technolgies are you talking about QDR IB, or something
PB> other than IB altogether, like 10 Gb Ethernet, or something all new and
PB> proprietary?
Cisco is dropping IB - probably to go for 10GE. It was a short time for Cisco offering IB solutions - bur probably they weren't very successful.
Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Wed Oct 1 14:22:57 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Wed, 1 Oct 2008 20:22:57 +0200
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <48E35A95.7020908@rri.sari.ac.uk>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
<48E35A95.7020908@rri.sari.ac.uk>
Message-ID: <61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl>
Well Tony,
Things are pretty simple for using SSI at your beowulf cluster.
The short summary:
a) openmosix is dead
b) open-ssi is still alive
Simple tip: go for open-ssi.
A bit longer explanation:
a) openmosix is dead. I can confirm that even wikipedia has that.
Openmosix was an Israeli project and therefore died when It's
Israeli main developer left, for whatever reason.
He has left a very clear statement that he no longer works on
openmosix.
What usually happens is that 1 or 2 enthusiasts then do an
effort to support it. If that doesn't work for you,
then consider it dead.
Using kernels 2.4.x is not realistic for todays quadcores. That was
the status in 2005, that is in 2008 still the status.
In 2005 if i remember well i used kernel 2.6.7 for a quad opteron
dual core.
Using kernel 2.4.x versus 2.6.x numa gave at a dual opteron dual core
a speedloss of 50% for my chessproggie.
50% is *a lot* to lose in speed.
Supporting a thing like openmosix requires a lot more than 1 guy who
in order to modify 3 bytes needs 1000 dollar.
What you see a lot is that open source projects get hijacked by
people who want to make cash out of it.
In the world i come from, computerchess, i remember already since
1988 that this happens every year several
times. The developers usually get really demotivated when they work
for hundreds of hours at their 'money project',
and then just make under a 1000 dollar; at the macdonalds you make
more money.
In short such projects usually die soon also, as there is no money
for them into it. Most nerds are social seen total robots.
b) open-ssi is there fore several distributions and actively
supported by several developers.
It is there, it works, it improves and it works for latest kernels
also (yes also 2.6.x).
In itself it would be GREAT if there is 1 open source project there,
as that joins forces more. My hope is that open-ssi
will work great for highend nic's also and slowly get to a phase that
all features work great.
OpenMosix nor openssi could migrate processes that work with shared
memory, a nerd feature i like personally a lot.
Just claiming that they use 'stolen' features like page migration is
like claiming that linux stole multithreading from unix;
it is a bad attempt to smear dirt just to earn a 1000 dollar.
It is not a reason to not use it. It is a bad attempt to spit at
these guys who donate time and their money without asking for payment.
Vincent
On Oct 1, 2008, at 1:10 PM, Tony Travis wrote:
> Vincent Diepeveen wrote:
>> I agree tony that paying for such crap is not very good idea.
>
> Hello, Vincent.
>
> I don't think MOSIX2 is crap!
>
> However, I don't like the idea of having to pay for 'updates'.
>
>> You might want to move to open-ssi in this case; the project is
>> alive and there is in theory work getting
>> performed on support for cards over infiniband as well.
>
> I have looked at OpenSSI, which uses the openMosix load-balancer,
> but process migration is more coarsely grained than in openMosix.
> Only the active pages of the user context of openMosix processes
> are migrated.
>
> I've been looking at alternatives and I think Kerrighed looks very
> promising but, in our hands, Kerrighed is very fragile: I've
> mentioned on this list before that if one Kerrighed node goes down
> you lose the entire cluster. We've been talking to Christine
> Morin's group at INRIA and they tell us that the next release of
> Kerrighed with be more robust:
>
> http://www.kerrighed.org
>
>> Most importantly is that you are gonna get more replies.
>
> Yes, thanks for yours :-)
>
>> Additionally the manner open-ssi implements shared memory is very
>> transparant; in principle on each write it migrates a page
>> to the node writing.
>> Maybe the only big lack of open-ssi is its limited support so far
>> for highend network cards.
>
> What bothers me about OpenSSI is that it's based on an open-sourced
> version HP's (now Compaq) discontinued commercial product "non-stop
> clusters for Unix". The OpenSSI project also came in for a lot of
> criticism from the openMosix community for stealing ideas, so my
> concerns about it might not be all that well founded ;-)
>
> The main reason I didn't use OpenSSI, previously, was that many
> features had not been implemented fully and, like Kerrighed, it
> wasn't really a viable option for a 'production' cluster even
> though it was interesting as a research project. What makes me take
> MOSIX2 seriously now is that it is a commercially supported
> 'product' with all the same virtues (and most of the vices) of
> openMosix.
>
> Tony.
> --
> Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
> and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
> tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
> mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From tegner at nada.kth.se Wed Oct 1 15:09:39 2008
From: tegner at nada.kth.se (Jon Tegner)
Date: Wed, 01 Oct 2008 21:09:39 +0200
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
Message-ID: <48E3CAF3.3070800@nada.kth.se>
There seem to be significant advantages using Scyld ClusterWare, I did
try it (Scyld?) many years ago (when it was free?) and I was impressed then.
However, when looking at penguincomputing.com I don't find any price
quotes. It seems - unless I miss something - one has to fill in a rather
lengthy form in order to get that information?
In order to consider the "Scyld solution" I think it would be good to
have at least an estimate of the price?
Regards,
/jon
Donald Becker wrote:
> On Wed, 1 Oct 2008, Bogdan Costescu wrote:
>
>
>> On Tue, 30 Sep 2008, Donald Becker wrote:
>>
>>> Ahhh, your first flawed assumption.
>>> You believe that the OS needs to be statically provisioned to the nodes.
>>> That is incorrect.
>>>
>> Well, you also make the flawed assumption that the best technical
>> solutions are always preferred. From my position I have seen many
>>
> ...
>
>> a solution like Scyld's limits the whole cluster to running one
>> distribution (please correct me if I'm wrong), while a solution with
>> node "images" allows mixing Linux distributions at will.
>>
>
> That's correct. Our model is that a "cluster" is a single system -- and a
> single install.
>
> That's for a good reason: To keep the simplicity and consistency of
> managing a single installation, you pretty much can have... only a single
> installation.
>
> There is quite a bit of flexibility. The system automatically detects the
> hardware and loads the correct kernel modules. Nodes can be specialized,
> including mounting different file systems and running different start-up
> scripts. But the bottom line is that to make the assertion that remote
> processes will run the same as local processes, they have to be running
> pretty much the same system.
>
> If you are running different distributions on nodes, you discard many of
> the opportunities of running a cluster. More importantly, it's much
> more knowledge- and labor-intensive to maintain the cluster while
> guaranteeing consistency.
>
>
>>> The only times that it is asked to do something new (boot, accept a
>>> new process) it's communicating with a fully installed, up-to-date
>>> master node. It has, at least temporarily, complete access to a
>>> reference install.
>>>
>> I think that this is another assumption that holds true for the Scyld
>> system, but there are situations where this is not true.
>>
>
> Yes, there are scenarios where you want a different model. But "connected
> during important events" is true for most clusters. We discard the
> ability for a node to boot and run independently in order to get the
> advantages of zero-install, zero-config consistent compute nodes.
>
>
>>> If you design a cluster system that installs on a local disk, it's
>>> very difficult to adapt it to diskless blades. If you design a
>>> system that is as efficient without disks, it's trivial to
>>> optionally mount disks for caching, temporary files or application
>>> I/O.
>>>
>> If you design a system that is flexible enough to allow you to use
>> either diskless or diskfull installs, what do you have to loose ?
>>
>
> In theory that sounds good. But historically changing disk-based
> installations to work on diskless machines has been very difficult, and
> the results unsatisfactory. Disk-based installations want to do selective
> installation based on the hardware present, and write/modify many links
> and configuration files on installation -- many more than they "need" to.
>
>
>> The same node "image" can be used in several ways:
>> - copied to the local disk and booted from there (where the copying
>> could be done as a separate operation followed by a reboot or it can
>> be done from initrd)
>> - used over NFS-root
>> - used as a ramdisk, provided that the node "image" is small enough
>>
>
> While memory follows the price-down capacity-up curve, we aren't quite to
> the point where holding a full OS distribution in memory is negligible.
> Most distributions (all the commercially interesting ones) are
> workstation-oriented, and the trade-off is "disk is under $1/GB, so we
> will install everything". It's foreseeable that holding an 8GB install
> image in memory will be trivial, but that will be a few years in the
> future, not today. And we will need better VM and PTE management to make
> it efficient.
>
>
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Wed Oct 1 16:03:38 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Wed, 1 Oct 2008 13:03:38 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E3BA69.2000405@ias.edu>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com>
<48E3BA69.2000405@ias.edu>
Message-ID: <20081001200338.GB32180@bx9.net>
On Wed, Oct 01, 2008 at 01:59:05PM -0400, Prentice Bisbal wrote:
> I'm sure you're right. It's my understanding that IB is a
> well-documented standard, so all IB cables should be created equally,
> but you know vendors, they'll say that their cables are more equal than
> others and refuse support if we're not using all Cisco kit.
Then buy from an IB vendor that isn't silly like that. QLogic and
Voltaire are choices. Is Cisco still re-selling QLogic switches?
Joe wrote:
> > Cisco does appear to be transitioning to *other* technologies. Again,
> > they aren't the only IB provider out there (seems such a shame, gobbling
> > up TopSpin and then effectively discarding them).
Apparently the main attraction of TopSpin was their virtualization
software. Cisco, of course, supports all the major technologies, so it
can be hard to tell if they really are transitioning to something
else. At a minimum Cisco will want to sell gateways for all
non-Ethernet technologies. After TopSpin was bought, Cisco continued
to sell gateways developed by TopSpin, but sold rebranded QLogic DDR
switches.
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From niftyompi at niftyegg.com Wed Oct 1 16:18:04 2008
From: niftyompi at niftyegg.com (NiftyOMPI Mitch)
Date: Wed, 1 Oct 2008 13:18:04 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <1848483940.20081001201911@gmx.net>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu>
<1848483940.20081001201911@gmx.net>
Message-ID: <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
On Wed, Oct 1, 2008 at 11:19 AM, Jan Heichler wrote:
> Hallo Prentice,
>
> Mittwoch, 1. Oktober 2008, meintest Du:
>
> PB> And what is the status of DDR? Are people still using it, or has it
>
> PB> already been replaced by QDR in the marketplace?
>
>
> DDR is still the most important Infiniband in the market. DDR provides
> enough bandwidth for most applications at the moment because most
> applications suffer from latency - not limited bandwidth. QDR is more
> interesting for building spine networks. And QDR might get more important if
> we see more cores in typical compute nodes.
>
>
> >> Cisco does appear to be transitioning to *other* technologies. Again,
>
> >> they aren't the only IB provider out there (seems such a shame, gobbling
>
> >> up TopSpin and then effectively discarding them).
>
>
> PB> What *other* technolgies are you talking about QDR IB, or something
>
> PB> other than IB altogether, like 10 Gb Ethernet, or something all new and
>
> PB> proprietary?
>
>
> Cisco is dropping IB - probably to go for 10GE. It was a short time for
> Cisco offering IB solutions - bur probably they weren't very successful.
>
Yes folks tell me that Cisco is backing away from the IB business -- all the
parts on the Cisco price book
that I know of were rebranded products and I suspect that some of their
providers were too happy to
sell directly to customers at a discount sometimes below the OEM price.
Perhaps more
importantly IB is not a router and management rich layer. i.e. It does not
facilitate all the routing
and management value add that Cisco focuses on for their bread and butter.
They are however well involved and commited in Open MPI which is agnostic to
the transport layer beyond the want to go fast part.
I know that QLogic has well specified and tested cables for sale. Call the
Qlogic "King of Prussia, Penn" office and tell them what you want. Gore and
Leoni make excellent cables as do others.
QDR is interesting... in all likelyhood the QDR game will be optical for any
link further away than a single rack. Once IB goes optical there will be a
lot of reason to install IB in machine rooms and campus sites
that are just out of reach today.
I should say that copper QDR is hard but not impossible. IMO optical has
advantages and once it becomes easy for the end user (site) to install
optical it should change the game.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From alscheinine at tuffmail.us Wed Oct 1 17:07:39 2008
From: alscheinine at tuffmail.us (Alan Louis Scheinine)
Date: Wed, 01 Oct 2008 16:07:39 -0500
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com>
<48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net>
<88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
Message-ID: <48E3E69B.4030202@tuffmail.us>
NiftyOMPI Mitch wrote
> QDR is interesting... in all likelyhood the
> QDR game will be optical for any link further away than a single rack.
> Once IB goes optical there will be a lot of reason to install IB in
> machine rooms and campus sites that are just out of reach today.
When will IB go optical at a reasonable price? Perhaps you are not
an expert but if you happen to have any pointers where we can learn more
about the time frame it would be useful. Just a few days ago heard
colleagues trying to figure-out the solution to connecting file system
hardware just a bit too far from a cluster in another building. Ethernet
and NFS would work but latency is a problem.
Best regards,
Alan
--
Alan Scheinine
5010 Mancuso Lane, Apt. 621
Baton Rouge, LA 70809
Email: alscheinine at tuffmail.us
Office phone: 225 578 0294
Mobile phone USA: 225 288 4176 [+1 225 288 4176]
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Wed Oct 1 17:27:11 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Wed, 01 Oct 2008 17:27:11 -0400
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
<48E2C6C1.60108@neuralbs.com>
Message-ID: <48E3EB2F.2070102@neuralbs.com>
Bogdan Costescu wrote:
> On Tue, 30 Sep 2008, Eric Thibodeau wrote:
>
>> This has given me much flexibility and a very fast path to upgrade
>> the nodes (LIVE!) since they would only need to be rebooted if I
>> changed the kernel. I can install/upgrade the node's environment by
>> simply chrooting into it and using the node's package manager and
>> utilities as if it were a regular system).
>
> Only the first is an advantage of using NFS-root; the second is shared
> by most methods that use a node "image".
More or less, NFS-root changes are "propagated" instantly, most other
approaches require a re-sync. Another way to see this, the NFS root
approach only does changes on the head node and changed files don't need
to be propagated and are accessed on a as-needed basis, this might have
significant impacts on large deployments....not that I suggest that they
use this approach ;)
> However random installations or modifications of configuration file
> within the chroot become very difficult to reproduce when you build
> the next node "image"
Document document document...which no one does...but document.
> - either scripting everything or using cfengine/puppet/etc. can save a
> lot of time in the long run, despite the initial effort to set up.
I'll take your word for it that they have a version tracking mechanism.
>> But I am in a special case where, if I break the cluster, I can fix
>> it quickly and I always have a backup copy of the boot "root" image
>> ready to switch to if my fiddling goes wrong.
> Why not keeping several "images" around and only point the nodes to
> mount the one considered current or "good" ?
Well, that's what I meant... probably not clearly. ;)
Eric
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From niftyompi at niftyegg.com Wed Oct 1 17:33:42 2008
From: niftyompi at niftyegg.com (NiftyOMPI Mitch)
Date: Wed, 1 Oct 2008 14:33:42 -0700
Subject: [Beowulf] precise synchronization of system clocks
In-Reply-To:
References:
<20081001020421.GA8126@compegg.wr.niftyegg.com>
Message-ID: <88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com>
On Wed, Oct 1, 2008 at 7:18 AM, Robert G. Brown wrote:
> On Tue, 30 Sep 2008, Nifty niftyompi Mitch wrote:
>
.....
> One site service of interest is AC power. A modern processor sitting
> in an idle state that then starts a well optimized loop will jump from
> a couple of watts to 100 watts in as many clocks as the set of pipelines
> is deep behind the instruction decode and instruction cache fill. A 1000
> processor (4000 cores) might jump from 4000 watts to 100000 watts in the
> blink of an eye (err did the lights blink). Buffer that dI/dT through
> the PS and it is less but still interesting on the mains which are
> synchronized.
>
Interesting. I never have seen the lights blink although I don't run
> synchronous computations. One wonders if the power supply capacitors
> (which should be quite large, I would think) don't soak up the
> transient, though, even on very large clusters. Also, I think that the
> power differential is smaller than you are allowing for -- I don't think
> most idle processors draw "no" power...
>
The dI/dT for processors can be quite high.
AMD Phenom? X4 Quad-Core is listed as a 140 watt part (thermal)
it is unlikely that all 450 million transistors are active in an idle
loop. Tom's Hardware list the idle power at 21 watts. The speed at
which a modern processor can go from idle to full power is astonishing.
The local on board power supply regulation must respond very quickly. The
delta
from 21 to 140 fit inside one half cycle of a 50/60 Hz AC mains service. So
inside
of one AC cycle the part can move from 21 to 140.... which is large when
multiplied by a 1000 node dual socket cluster. I do know of clusters and
labs of workstations that power on hosts and disks in sequence to limit the
startup power surge. Lots of us have been at it long enough to know that
induction motors like elevators, refrigeration compressors and even vacuums
can hit the mains
hard enough to trigger errors. My home vacuum does dim the lights a little
bit.
In normal practice I doubt that this is an issue but synchronization in the
extreme
is interesting in its details and side effects.
--
NiftyOMPI
T o m M i t c h e l l
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From andrew at moonet.co.uk Wed Oct 1 18:22:08 2008
From: andrew at moonet.co.uk (andrew holway)
Date: Wed, 1 Oct 2008 23:22:08 +0100
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E39782.4020006@ias.edu>
References: <48E39782.4020006@ias.edu>
Message-ID:
> Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
I don't think front side busses are fast enough for QDR yet.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpatienc at cisco.com Wed Oct 1 19:58:51 2008
From: rpatienc at cisco.com (Rob Patience)
Date: Wed, 1 Oct 2008 16:58:51 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <20081001200338.GB32180@bx9.net>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com>
<48E3BA69.2000405@ias.edu> <20081001200338.GB32180@bx9.net>
Message-ID: <1F19606C-6310-40F9-B2F3-AC32671D620E@cisco.com>
Only the larger switches were from QLogic (144 and 288 ports) all
other switches were made by TopSpin/Cisco directly. Cisco is not
leaving the HPC space only IB.
Btw: you can still buy cables from Cisco. But as mentioned, all
cables come from 3rd party vendors.
~rob
On Oct 1, 2008, at 1:03 PM, Greg Lindahl wrote:
> On Wed, Oct 01, 2008 at 01:59:05PM -0400, Prentice Bisbal wrote:
>
>> I'm sure you're right. It's my understanding that IB is a
>> well-documented standard, so all IB cables should be created equally,
>> but you know vendors, they'll say that their cables are more equal
>> than
>> others and refuse support if we're not using all Cisco kit.
>
> Then buy from an IB vendor that isn't silly like that. QLogic and
> Voltaire are choices. Is Cisco still re-selling QLogic switches?
>
> Joe wrote:
>
>>> Cisco does appear to be transitioning to *other* technologies.
>>> Again,
>>> they aren't the only IB provider out there (seems such a shame,
>>> gobbling
>>> up TopSpin and then effectively discarding them).
>
> Apparently the main attraction of TopSpin was their virtualization
> software. Cisco, of course, supports all the major technologies, so it
> can be hard to tell if they really are transitioning to something
> else. At a minimum Cisco will want to sell gateways for all
> non-Ethernet technologies. After TopSpin was bought, Cisco continued
> to sell gateways developed by TopSpin, but sold rebranded QLogic DDR
> switches.
>
> -- greg
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rgb at phy.duke.edu Thu Oct 2 00:14:06 2008
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 2 Oct 2008 00:14:06 -0400 (EDT)
Subject: [Beowulf] precise synchronization of system clocks
In-Reply-To: <88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com>
References:
<20081001020421.GA8126@compegg.wr.niftyegg.com>
<88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com>
Message-ID:
On Wed, 1 Oct 2008, NiftyOMPI Mitch wrote:
> The dI/dT for processors can be quite high.
> AMD Phenom??? X4 Quad-Core is listed as a 140 watt part (thermal)
> it is unlikely that all 450 million transistors are active in an idle
> loop. Tom's Hardware list the idle power at 21 watts. The speed at
> which a modern processor can go from idle to full power is astonishing.
> The local on board power supply regulation must respond very quickly. The
> delta
> from 21 to 140 fit inside one half cycle of a 50/60 Hz AC mains service. So
> inside
> of one AC cycle the part can move from 21 to 140.... which is large when
> multiplied by a 1000 node dual socket cluster. I do know of clusters and
> labs of workstations that power on hosts and disks in sequence to limit the
> startup power surge. Lots of us have been at it long enough to know that
> induction motors like elevators, refrigeration compressors and even vacuums
> can hit the mains
> hard enough to trigger errors. My home vacuum does dim the lights a little
> bit.
I understand inductive surge when powering up, I understand in detail
browning out a primary power transformer, but I think those are
different issues and irrelevant here.
So far, using my trusty Kill-a-Watt on real world nodes, I haven't seen
more than a 30% differential draw loaded to unloaded. Large parts of
the CPU require power at all times to function. Memory, for example,
both on and offboard. Nearly everything inside a computer has a
nontrivial idle draw, plus (sure) peak draw when it or one of its
subsystems are in use.
Exceptions are modern laptops -- with variable speed clocks, they draw
much less idling than they do at speed, in part because power (idle or
otherwise) IS very nearly proportional to CPU clock in at least parts of
the system. And I don't really know how the latest designs do in this
regard -- but there is a tendency to design the bleeding edge
PERFORMANCE CPUs to work at "constant heat", as a major rate limiting
factor in CPU design is getting rid of heat from the package. It's one
reason they don't just crank the clock to infinity and beyond -- not the
only one, but a major one.
Multicores, of course, may function like hybrid cars, and somehow run
more nearly idle when they are idle. But I'd have to hear from someone
who slapped a KaW on an actual system and clocked it from idle (solidly
post-boot, running the OS, at idle "equilibrium") to loaded (running
flat out on e.g. a benchmark suite that loads all cores and/or the
memory etc.). Has anyone actually done this and observed (say) a 2 or 3
to 1 increase in power draw loaded to idle? 50W idle to 200W loaded in
1 second? 150W idle to 200W loaded is more like what I've seen...
> In normal practice I doubt that this is an issue but synchronization in the
> extreme is interesting in its details and side effects.
I completely agree with this, both parts. Although if one IS bumping
from 50->200W "instantly" on not even an entire cluster but just all the
nodes on a single circuit, that's popping over a KW on a 20A line --
ballpark where one MIGHT see something inductive (although as I said,
probably nothing that the power supply capacitor(s) cannot buffer,
although I'm too tired to compute the number of joules (watt-seconds)
one can probably deliver and what RC probably is, etc). Popping
multiple (as in 10+) KW in less than a 60 Hz cycle would very likely be
hard on the primary, no doubt about it.
rgb
--
NiftyOMPI
T o m M i t c h e l l
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 2 05:05:30 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 2 Oct 2008 10:05:30 +0100
Subject: [Beowulf] Linux Magazine - What He Said
Message-ID: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
I just read Douglas Eadline's article on Linux Magazine, entitled "What He
Said"
http://www.linux-mag.com/id/7087
Very thought provoking article, and took me back to thinking about genetic
algorithms,
a subject I flirted with 20 years ago. I didn't find it worthwhile on a
Sparc1 system with a whopping 2 Mbytes of RAM.
I guess I should encourage responses to be made on the Linux Mag site.
John Hearns
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From ajt at rri.sari.ac.uk Thu Oct 2 05:25:33 2008
From: ajt at rri.sari.ac.uk (Tony Travis)
Date: Thu, 02 Oct 2008 10:25:33 +0100
Subject: [Beowulf] Re: MOSIX2
In-Reply-To: <61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl>
References: <039801c92260$ab3630f0$1f9886a5@its99fd46g>
<48E1EDE3.2080205@rri.sari.ac.uk>
<48E35A95.7020908@rri.sari.ac.uk>
<61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl>
Message-ID: <48E4938D.202@rri.sari.ac.uk>
Vincent Diepeveen wrote:
> [...]
> Supporting a thing like openmosix requires a lot more than 1 guy who in
> order to modify 3 bytes needs 1000 dollar.
Hello, Vincent.
I've been involved in the openMosix project from the start, and I still
use it. My comment about the $1,000 for 'updates' concerned the cost an
'academic' update and support subscription to MOSIX2, which has nothing
to do with openMosix. I think that is quite a reasonable cost, actually,
and I dodn't criticise people who want to pay for support. It's not the
cost that bothers me most, but the restrictions of the MOSIX2 licence.
> [...]
> In itself it would be GREAT if there is 1 open source project there, as
> that joins forces more. My hope is that open-ssi
> will work great for highend nic's also and slowly get to a phase that
> all features work great.
OK, maybe I should have another look at the most recent OpenSSI...
> OpenMosix nor openssi could migrate processes that work with shared
> memory, a nerd feature i like personally a lot.
Actually there is a version of openMosix that can migrate programs using
shared memory. Unfortunately, when I tried it, processes would migrate
away from the 'home' node but wouldn't migrate back! This is a problem
for openMosix, because processes have to migrate back to the 'home' node
to make system calls. One of my (small) contributions to openMosix was
to patch it to avoid migrations just to read the time. This is slightly
relevant to another thread on the Beowulf list, because reading the time
locally on a node to avoid the process migration overhead requires that
the nodes all run NTP otherwise you get problems with clock skew...
> Just claiming that they use 'stolen' features like page migration is
> like claiming that linux stole multithreading from unix;
> it is a bad attempt to smear dirt just to earn a 1000 dollar.
>
> It is not a reason to not use it. It is a bad attempt to spit at these
> guys who donate time and their money without asking for payment.
Sadly, there was a lot of bad feeling between the openMosix and OpenSSI
communities and I should not have tried to make light of it. In fact it
is a compliment to openMosix that the load-balancer algorithm was used
in OpenSSI, not an insult, and BTW I'm one of those guys who donated
their time without asking for payment.
I did put a wry smile after my comment ;-)
Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:ajt at rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Thu Oct 2 08:03:30 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Thu, 2 Oct 2008 14:03:30 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
Message-ID:
On Wed, 1 Oct 2008, Donald Becker wrote:
> That's correct. Our model is that a "cluster" is a single system --
> and a single install.
That's the idea that I've also started with, almost 10 years ago ;-)
Not using Beo*/bproc, but NFS-root which allowed a single install in
the node "image" to be used on all nodes - although you'd probably
call this 2 system installs (the master node itself and the node
"image"). But over the course of the years I have changed my mind...
> If you are running different distributions on nodes, you discard
> many of the opportunities of running a cluster. More importantly,
> it's much more knowledge- and labor-intensive to maintain the
> cluster while guaranteeing consistency.
It indeed requires more work, however in some cases it cannot be
avoided. From my own experience: a quantum chemistry program was
distributed some 5 years ago as a binary statically compiled on RH9 or
RHEL3 (kernel 2.4 based) with MPICH included. This meant that when I
wanted to switch to running a 2.6 kernel this program could not run
anymore so some of the nodes had to be kept to an older distribution
until a newer program version could be obtained (that took about a
year); it also meant that whenever there were discussions about using
higher performance interconnects than GigE, this software's users were
insisting on buying more nodes rather than a faster interconnect. This
situation has caused both technical and administrative issues and the
possibility of running different distributions has solved all of them
easily.
Having the possibility to run several distributions side-by-side
requires spending some effort in organizing the other installed
software, normally shared through NFS or a parallel FS to the nodes.
But once you make the jump from 1 to 2, you might as well make it from
1 to many.
This leads me to observe that we have non-similar points of view: you
are a maker of a cluster-oriented distribution, trying to promote it
and its underlying ideas (which are fine ideas, no question about that
:-)), and sure that it works because it was bought and used
successfully. I, on the other hand, have to find solutions to keep the
scientists productive (whatever productive means ;-)) and to keep them
as far as possible from the system details so that they can
concentrate on their work. So it's not surprising that we come to
different conclusions - at least they sustain an interesting
discussion :-)
I would be interested to hear Mark Hahn's opinion on this, as from how
he presented himself to this list it seemed to me that he is in a very
similar position to mine: supporting a variety of users with a variety
of needs. But others should not feel left out, write your opinions as
well ;-)
> Most distributions (all the commercially interesting ones) are
> workstation-oriented
I don't really agree with this statement (looking at RHEL and SLES),
but anyone who installs a workstation-oriented distribution on a
cluster node gets what (s)he pays for :-) I have seen very recently
(identity hidden to protect the guilty ;-)) such a node "image" which
contained OpenOffice - to be fair, it was used via NFS-root so it
wasn't wasting node memory, only master disk space...
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Thu Oct 2 08:18:30 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Thu, 2 Oct 2008 14:18:30 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <48E3EB2F.2070102@neuralbs.com>
References:
<48E2C6C1.60108@neuralbs.com>
<48E3EB2F.2070102@neuralbs.com>
Message-ID:
On Wed, 1 Oct 2008, Eric Thibodeau wrote:
> the NFS root approach only does changes on the head node and changed
> files don't need to be propagated and are accessed on a as-needed
> basis, this might have significant impacts on large deployments
NFS-root doesn't scale too well, the implementation of NFS in Linux is
quite chatty.
> I'll take your word for it that they have a version tracking mechanism.
Take my word that if you're going into larger installations with the
least amount of non-homogeneity you'll want to at least read about, if
not use, such mechanisms :-)
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 2 08:25:17 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 2 Oct 2008 13:25:17 +0100
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
Message-ID: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
2008/10/1 Donald Becker
>
>
> It's foreseeable that holding an 8GB install
> image in memory will be trivial, but that will be a few years in the
> future, not today. And we will need better VM and PTE management to make
> it efficient.
>
>
>
Hmmmm.... can I forsee Puppy Linux HPC Edition ????
http://www.puppylinux.org/
Being half serious here, is it worth trying to get one of these slimmed-down
distros to the state where it will run an HPC job?
Oh, and in addition to a barebones install for our contemplative cluster,
they have a onebones install which cuts out the GUI (cue more dog puns).
http://www.puppylinux.com/pfs/ looks interesting...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From smulcahy at aplpi.com Thu Oct 2 08:56:41 2008
From: smulcahy at aplpi.com (stephen mulcahy)
Date: Thu, 02 Oct 2008 13:56:41 +0100
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
Message-ID: <48E4C509.3000407@aplpi.com>
John Hearns wrote:
> Hmmmm.... can I forsee Puppy Linux HPC Edition ????
> http://www.puppylinux.org/
>
>
> Being half serious here, is it worth trying to get one of these
> slimmed-down distros to the state where it will run an HPC job?
> Oh, and in addition to a barebones install for our contemplative
> cluster, they have a onebones install which cuts out the GUI (cue more
> dog puns).
Is there that much of a difference between Puppy and a minimal Debian
where you install only the "standard server" task (I understand Puppy is
a Debian derivative, apologies if this is incorrect)?
Or am I swinging my Debian swiss-army chainsaw indiscriminately here?
-stephen
--
Stephen Mulcahy Applepie Solutions Ltd. http://www.aplpi.com
Registered in Ireland, no. 289353 (5 Woodlands Avenue, Renmore, Galway)
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Thu Oct 2 09:28:38 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Thu, 2 Oct 2008 15:28:38 +0200
Subject: [Beowulf] Linux Magazine - What He Said
In-Reply-To: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
Message-ID:
To comment a bit on the article:
>What if a high level programing description language was developed.
Note I did not say programming language.
>This description language would allow you to ?describe? what you
needed to do and not how to do it (as
>discussed before). This draft description would then be presented
to an AI based clarifier which would examine
>the description, look for inconsistencies or missing information
and work with the programmer to create a formal
>description of the problem. At that point the description is turned
over to a really smart compiler that could target
>a particular hardware platform and produce the needed optimized
binaries.
>Perhaps a GA could be thrown in to help optimize everything.
The typical average PHD way of approaching problems they need to solve:
Suppose you need to solve a huge problem A at a major parallel machine.
We throw the problem A into a blackbox B giving output A'.
Now our fantastic new brilliant method/algorithm comes into place
that works on A' what our paper is about,
and we solve the problem with that.
I see that scenario a lot in all kind of different variations.
A great example of it is:
We invent algorithm f( x )
Now in our program we have
if( magicfunction(x) && f(x) )
then printf("problem solved. jippie i got my research done.\n");
They test it at 1 case C and shout victory.
typical AI way of doing things. There is so little AI dudes who do
more than fill 50 pages of paper a year with ideas that keep untested,
and because of never testing their understanding of the problem never
advances and the next guy still shows up with the same untested
solution. So the above guy who actually is *doing* something and
testing something, already gets cheered for (with some reasons).
But of course magicfunction(x) only works for their case C.
There is no statistic significance.
The number of AI guys who test their algorithm/method at state of the
art software, be it selfwritten or from someone else,
AND test is statistical significant without using some flawed testset
where magicfunction(x) is always true, those guys you can
really count on 2 hands the past 30 years.
In parallel AI it is even worse in fact. There is for example just 2
chessprograms that scale well (so without losing first factor 50 to
the parallel
search frame) at supercomputers. There is some fuzz now about go
programs using UCT/Monte Carlo and a few other algorithms
combined; but these random algorithms miss such simple tactics, which
for their computing
power should be easy to not miss, that they really should think out a
better way to search there parallel.
Each AI solution that is not embarrassingly parallel is so difficult
to parallellize well, so not losing a factor 50,
that it is just real hard to make generic frameworks that work real
efficient. Such a framework would pose a solution for only 1
program; as soon as you improve 1 year later the program with a new
enhancement the entire parallel framework might need to get
rewritten from scratch again, to not again lose some factors in speed
in scaling.
The commercial way most AI guys look to the parallel problem is
therefore real simple:
"can i get faster than i am at a PC, as it is a government
machine anyway,
i didn't pay for efficient usage of it, i just want to be faster
than my home PC
without too much effort".
I'm not gonna fight that lemma.
However a generic framework that works like that is not gonna get
used of course. It just speeds you up so much to make a custom
parallel solution, that everyone is doing it.
Besides, the hardware is that expensive, that it's worth doing it.
>This process sounds like it would take a lot of computing
resources. Guess what? We have that.
>Why not throw a cluster at this problem. Maybe it would take a week
to create a binary,
>but it would be cluster time and not your time. There would be no
edit/make/run cycle
>because the description tells the compiler what the program has to
do. The minutia
>(or opportunities for bugs) of programming whether it be serial or
parallel would be
>handled by the compiler. Talk about a killer application.
On Oct 2, 2008, at 11:05 AM, John Hearns wrote:
> I just read Douglas Eadline's article on Linux Magazine, entitled
> "What He Said"
> http://www.linux-mag.com/id/7087
>
> Very thought provoking article, and took me back to thinking about
> genetic algorithms,
> a subject I flirted with 20 years ago. I didn't find it worthwhile
> on a Sparc1 system with a whopping 2 Mbytes of RAM.
>
> I guess I should encourage responses to be made on the Linux Mag site.
>
> John Hearns
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From james.p.lux at jpl.nasa.gov Thu Oct 2 09:37:58 2008
From: james.p.lux at jpl.nasa.gov (Lux, James P)
Date: Thu, 2 Oct 2008 06:37:58 -0700
Subject: [Beowulf] precise synchronization of system clocks
In-Reply-To:
Message-ID:
Rgb wrote:
>
> I understand inductive surge when powering up, I understand in detail
> browning out a primary power transformer, but I think those are
> different issues and irrelevant here.
Inductive surge -> magnetizing current in large iron core inductors (depends
on where you are in line frequency cycle at "switch on")
Sag from overload -> impedance (both R and L) in transformer and wires from
transformer to load. 2% voltage drop is the NEC guideline for "in premises
wires".. From panel to load. The voltage at the utility entrance could
probably be +/- 5% at any given time.
>
> So far, using my trusty Kill-a-Watt on real world nodes, I haven't seen
> more than a 30% differential draw loaded to unloaded. Large parts of
> the CPU require power at all times to function. Memory, for example,
> both on and offboard. Nearly everything inside a computer has a
> nontrivial idle draw, plus (sure) peak draw when it or one of its
> subsystems are in use.
Very much true. DRAM needs refresh, for instance.
>
> Exceptions are modern laptops -- with variable speed clocks, they draw
> much less idling than they do at speed, in part because power (idle or
> otherwise) IS very nearly proportional to CPU clock in at least parts of
> the system. And I don't really know how the latest designs do in this
>
> Multicores, of course, may function like hybrid cars, and somehow run
> more nearly idle when they are idle. But I'd have to hear from someone
> who slapped a KaW on an actual system and clocked it from idle (solidly
> post-boot, running the OS, at idle "equilibrium") to loaded (running
> flat out on e.g. a benchmark suite that loads all cores and/or the
> memory etc.). Has anyone actually done this and observed (say) a 2 or 3
> to 1 increase in power draw loaded to idle? 50W idle to 200W loaded in
> 1 second? 150W idle to 200W loaded is more like what I've seen...
Don't forget that the power supply efficiency drops dramatically when DC
load drops, too. They don't spend a penny more on sophisticated design than
required to get that "energy star" rating, and that has more to do with
having a good "low power hibernate" mode than good efficiency at 25% load.
>
>> In normal practice I doubt that this is an issue but synchronization in the
>> extreme is interesting in its details and side effects.
>
> I completely agree with this, both parts. Although if one IS bumping
> from 50->200W "instantly" on not even an entire cluster but just all the
> nodes on a single circuit, that's popping over a KW on a 20A line --
> ballpark where one MIGHT see something inductive (although as I said,
> probably nothing that the power supply capacitor(s) cannot buffer,
> although I'm too tired to compute the number of joules (watt-seconds)
> one can probably deliver and what RC probably is, etc). Popping
> multiple (as in 10+) KW in less than a 60 Hz cycle would very likely be
> hard on the primary, no doubt about it.
If one considers that a single wire is about 1 uH/meter (typical electrical
wiring will be much less, because it's a pair, with currents flowing
opposite directions), the series L might be a few tens of uH. At, say, 20
A, there's just not much energy stored there.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rgb at phy.duke.edu Thu Oct 2 10:41:24 2008
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 2 Oct 2008 10:41:24 -0400 (EDT)
Subject: [Beowulf] precise synchronization of system clocks
In-Reply-To:
References:
Message-ID:
On Thu, 2 Oct 2008, Lux, James P wrote:
> If one considers that a single wire is about 1 uH/meter (typical electrical
> wiring will be much less, because it's a pair, with currents flowing
> opposite directions), the series L might be a few tens of uH. At, say, 20
> A, there's just not much energy stored there.
I was thinking in terms of load on transformers, but yeah, I forgot
(again) that switching power supplies ain't got no transformers.
Voltage regulators and MAYBE UPS have transformers, but they also have
bloody damn big capacitors. I'm just not used to a transformer-free
world.
Back in the very old days we had power problems in our server room (that
I think might have been connected to people using really big physics
apparatus in the building) and we bought a honker power conditioner to
run a subset of our systems. If one set up a monitor within two meters
of the sucker, the CRT fuzzed and distorted -- the rapidly varying field
was strong enough to deflect electrons at two meters. We kept it far
far away from our backup tapes...;-)
rgb
--
Robert G. Brown Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From jamesb at loreland.org Thu Oct 2 11:37:50 2008
From: jamesb at loreland.org (James Braid)
Date: Thu, 2 Oct 2008 16:37:50 +0100
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
<48E2C6C1.60108@neuralbs.com>
<48E3EB2F.2070102@neuralbs.com>
Message-ID: <25446eb90810020837v6d8e91bdt6dc36a634dca18b5@mail.gmail.com>
2008/10/2 Bogdan Costescu :
> On Wed, 1 Oct 2008, Eric Thibodeau wrote:
>
>> the NFS root approach only does changes on the head node and changed files
>> don't need to be propagated and are accessed on a as-needed basis, this
>> might have significant impacts on large deployments
>
> NFS-root doesn't scale too well, the implementation of NFS in Linux is quite
> chatty.
It's scaled great in our experience. We run 1000+ machines off NFS
root, running lots of large third-party applications from there as
well.
The NFS servers are just a pair of Linux servers, nothing fancy.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From peter.st.john at gmail.com Thu Oct 2 12:41:13 2008
From: peter.st.john at gmail.com (Peter St. John)
Date: Thu, 2 Oct 2008 12:41:13 -0400
Subject: [Beowulf] Linux Magazine - What He Said
In-Reply-To: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
Message-ID:
John,
After I first thought up my nutty GA scheme, I was then astonished by the
John Holland Scientific American artifcle. I was aghast that anyone could
even imagine thinking along those lines with 1960's hardware, as he had
done.
I got my first working version done on a 386 (daughtercard on a 286
motherboard) with one and a half MB.
Peter
On 10/2/08, John Hearns wrote:
>
> I just read Douglas Eadline's article on Linux Magazine, entitled "What He
> Said"
> http://www.linux-mag.com/id/7087
>
> Very thought provoking article, and took me back to thinking about genetic
> algorithms,
> a subject I flirted with 20 years ago. I didn't find it worthwhile on a
> Sparc1 system with a whopping 2 Mbytes of RAM.
>
> I guess I should encourage responses to be made on the Linux Mag site.
>
> John Hearns
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 2 12:54:13 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 2 Oct 2008 17:54:13 +0100
Subject: [Beowulf] Linux Magazine - What He Said
In-Reply-To:
References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
Message-ID: <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com>
2008/10/2 Peter St. John
> John,
> After I first thought up my nutty GA scheme, I was then astonished by the
> John Holland Scientific American artifcle. I was aghast that anyone could
> even imagine thinking along those lines with 1960's hardware, as he had
> done.
>
> I believe I read the same article.
In fact, the first algorithm I tried was simulated annealing (I know this
does not equal a GA). It swapped so horrendously that I was discouraged, and
the thing took all night to run even for a simple 2D case (I was working on
radiation therapy planning).
I guess I should have been smarter and worked out how to do it within the
constraints of RAM that I had.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From peter.st.john at gmail.com Thu Oct 2 14:06:55 2008
From: peter.st.john at gmail.com (Peter St. John)
Date: Thu, 2 Oct 2008 14:06:55 -0400
Subject: [Beowulf] Linux Magazine - What He Said
In-Reply-To: <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com>
References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
<9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com>
Message-ID:
John,
When I tried to ressurect my thing a couple years ago, I realized my
original code was all wrong in trading time for space (plenty of time on the
386, then SunOS servers; not enough space, but new machine had plenty of
unused RAM). I thought some about redesigning to reverse the trade-off,
which would be helpful, but I'm sure it would not just be easier, but more
effective, to run it on a cluster (many nodes, not much ram per node needed,
and any nonzero amount of communication sufficient, but more can be
usefull).
Peter
On 10/2/08, John Hearns wrote:
>
>
>
> 2008/10/2 Peter St. John
>
>> John,
>> After I first thought up my nutty GA scheme, I was then astonished by the
>> John Holland Scientific American artifcle. I was aghast that anyone could
>> even imagine thinking along those lines with 1960's hardware, as he had
>> done.
>>
>> I believe I read the same article.
> In fact, the first algorithm I tried was simulated annealing (I know this
> does not equal a GA). It swapped so horrendously that I was discouraged, and
> the thing took all night to run even for a simple 2D case (I was working on
> radiation therapy planning).
> I guess I should have been smarter and worked out how to do it within the
> constraints of RAM that I had.
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From gdjacobs at gmail.com Thu Oct 2 14:42:58 2008
From: gdjacobs at gmail.com (Geoff Jacobs)
Date: Thu, 02 Oct 2008 13:42:58 -0500
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <48E4C509.3000407@aplpi.com>
References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
<48E4C509.3000407@aplpi.com>
Message-ID: <48E51632.8090001@gmail.com>
stephen mulcahy wrote:
> John Hearns wrote:
>> Hmmmm.... can I forsee Puppy Linux HPC Edition ????
>> http://www.puppylinux.org/
>>
>>
>> Being half serious here, is it worth trying to get one of these
>> slimmed-down distros to the state where it will run an HPC job?
>> Oh, and in addition to a barebones install for our contemplative
>> cluster, they have a onebones install which cuts out the GUI (cue more
>> dog puns).
>
> Is there that much of a difference between Puppy and a minimal Debian
> where you install only the "standard server" task (I understand Puppy is
> a Debian derivative, apologies if this is incorrect)?
>
> Or am I swinging my Debian swiss-army chainsaw indiscriminately here?
>
> -stephen
>
Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast.
This is an interesting topic, though. How much difference does shrinking
the size of a kernel build make in improving the performance of a
tightly coupled lockstep algorithm scaled to hundreds, or thousands of
nodes? Anything in the literature which is directly comparable, not just
the ASCI Q paper?
--
Geoffrey D. Jacobs
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From xclski at yahoo.com Thu Oct 2 15:10:40 2008
From: xclski at yahoo.com (Ellis Wilson)
Date: Thu, 2 Oct 2008 12:10:40 -0700 (PDT)
Subject: [Beowulf] Linux Magazine - What He Said
Message-ID: <217759.53687.qm@web37906.mail.mud.yahoo.com>
In the article:
"What if a high level programing description language was developed.
Note I did not say programming language. This description language would
allow you to ?describe? what you needed to do and not how to do it (as
discussed before)."
I would ask then, how does one "describe what you need to do"? As a
brief (and heinously simplistic) example, let us say that we wanted to
replace a specific string in a long text file with another string. In
assembler, this would be a heinous and explicit long process where we
tell it exactly what we are doing. One could even argue at the C level
we have a fair amount of control, but once we hit fairly high-level
programming languages one merely can say
text.replaceAll(oldstring,newstring); and you are done. You have told
the program what you want done, not how to do it. Would I call Java,
C#, etc. "Programming Description Languages"? No. Therefore I wouldn't
call an even higher level HPC language a description language either.
In the article:
"This draft description would then be presented to an AI based clarifier
which would examine the description, look for inconsistencies or missing
information and work with the programmer to create a formal description
of the problem."
Sounds like regular programming in an intolerant IDE with fancy terminology.
In the article:
"At that point the description is turned over to a really smart compiler
that could target a particular hardware platform and produce the needed
optimized binaries. Perhaps a GA could be thrown in to help optimize
everything."
Later on it is also mentioned that "Maybe it would take a week to create
a binary, but it would be cluster time and not your time", where in
reality with those really troublesome (useful) problems there are truly
terribly long running times. With a GA (which produces eons more bad
solutions than good) we would not only have to ascertain the fitness of
the really nice solution (for those useful problems it could take a week
or more at fastest) but also the fitness of the really really poor
solution that swaps out constantly and computes redundantly. That could
take years...
The basic premise of the GA for code is Genetic Programming or an
Evolutionary Algorithm, and so with these the same problems exist - bad
solutions that monopolize time on the cluster.
Compilers will eventually be entirely AI (though I doubt I will see it)
and when they are, singularity will have already happened and infinite
resources will be available since designing hardware is naturally more
space constrained than software. All I'm saying is for right now, we
are making the most of what we have without involving AI that
extensively in our programming.
Just my opinions, and no hard feelings towards Doug. Typically I enjoy
thoroughly his articles.
Ellis
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 2 15:15:31 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 2 Oct 2008 20:15:31 +0100
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <48E51632.8090001@gmail.com>
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
<48E4C509.3000407@aplpi.com> <48E51632.8090001@gmail.com>
Message-ID: <9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com>
>
> Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast.
>
> The FAQ on the site says its a Slackware derivative, if I'm not wrong.
What goes around comes around I guess :-) Maybe those kipper ties from the
70s will be back too.
Actually, and here I toss in a handgrenade, if we are considering a Damn
Small Puppy PXE bootable Linux in RAM, what difference does the distro
make, so long as the GLIBC/maths libraries/MPI libraries are the
appropriate ones.
I guess that statement goes for any Linux install really.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From dgs at slac.stanford.edu Thu Oct 2 14:47:48 2008
From: dgs at slac.stanford.edu (David Simas)
Date: Thu, 2 Oct 2008 11:47:48 -0700
Subject: [Beowulf] Linux Magazine - What He Said
In-Reply-To:
References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com>
<9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com>
Message-ID: <20081002184748.GA29815@horus.slac.stanford.edu>
> When I tried to ressurect my thing a couple years ago, I realized my
> original code was all wrong in trading time for space (plenty of time on the
> 386, then SunOS servers; not enough space, but new machine had plenty of
> unused RAM). I thought some about redesigning to reverse the trade-off,
> which would be helpful, but I'm sure it would not just be easier, but more
> effective, to run it on a cluster (many nodes, not much ram per node needed,
> and any nonzero amount of communication sufficient, but more can be
> usefull).
In case you don't know about PGAPack:
http://www-fp.mcs.anl.gov/CCST/research/reports_pre1998/comp_bio/stalk/pgapack.html
It's a genetic algorithm with MPI support. I've used the serial version,
and it works great. I made a half-effort at getting the MPI version
working, without success.
DGS
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From YXU11 at PARTNERS.ORG Thu Oct 2 16:09:36 2008
From: YXU11 at PARTNERS.ORG (Xu, Jerry)
Date: Thu, 2 Oct 2008 16:09:36 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <200810021900.m92J05RP012792@bluewest.scyld.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
Message-ID: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Hello,
Currently I generate nearly one TB data every few days and I need to pass it
along enterprise network to the storage center attached to my HPC system, I am
thinking about compressing it (most tiff format image data) as much as I can, as
fast as I can before I send it crossing network ... So, I am wondering whether
anyone is familiar with any hardware based accelerator, which can dramatically
improve the compressing procedure.. suggestion for any file system architecture
will be appreciated too.. I have couple of contacts from some vendors but not
sure whether it works as I expected, so if anyone has experience about it and
want to share, it will be really appreciated !
Thanks,
Jerry
Jerry Xu PhD
HPC Scientific Computing Specialist
Enterprise Research Infrastructure Systems (ERIS)
Partners Healthcare, Harvard Medical School
http://www.partners.org
The information transmitted in this electronic communication is intended only
for the person or entity to whom it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or other
use of or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you received this
information in error, please contact the Compliance HelpLine at 800-856-1983 and
properly dispose of this information.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From niftyompi at niftyegg.com Thu Oct 2 17:29:44 2008
From: niftyompi at niftyegg.com (NiftyOMPI Mitch)
Date: Thu, 2 Oct 2008 14:29:44 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E3E69B.4030202@tuffmail.us>
References: <48E39782.4020006@ias.edu>
<48E3B1F4.3000702@scalableinformatics.com>
<48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net>
<88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
<48E3E69B.4030202@tuffmail.us>
Message-ID: <20081002212944.GA6110@compegg.wr.niftyegg.com>
On Wed, Oct 01, 2008 at 04:07:39PM -0500, Alan Louis Scheinine wrote:
> NiftyOMPI Mitch wrote
>> QDR is interesting... in all likelyhood the
>> QDR game will be optical for any link further away than a single rack.
>> Once IB goes optical there will be a lot of reason to install IB in
> > machine rooms and campus sites that are just out of reach today.
>
> When will IB go optical at a reasonable price? Perhaps you are not
> an expert but if you happen to have any pointers where we can learn more
> about the time frame it would be useful. Just a few days ago heard
> colleagues trying to figure-out the solution to connecting file system
> hardware just a bit too far from a cluster in another building. Ethernet
> and NFS would work but latency is a problem.
I have no good information and I do not know what a reasonable price is to you.
If you wish to connect to a cluster in another building
with optical links and IB there are some optical solutions today.
Without "a lot of links" (=money) bandwidth maps will be lumpy.
Building to building links and beyond solutions will be expensive
many have active boxes switch-Cu---optical---Cu-switch.
For the current solutions scan the vendor lists from recent Supercomputer
Shows and IB interoperability events.
Floor to floor and room to room solutions like the Intel Optical IB
cables are close to par with copper when you multiply by the link length.
If you want good information put out a sane RFP and see what the vendors
can come up with. Most importantly describe what you want solved
and see what solutions are offered. Building to building clustering
just seems hard to me but not all clustering requirements are equal.
--
T o m M i t c h e l l
Found me a new hat, now what?
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Thu Oct 2 17:40:31 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Thu, 02 Oct 2008 17:40:31 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID: <48E53FCF.3020008@scalableinformatics.com>
Xu, Jerry wrote:
> Hello,
>
> Currently I generate nearly one TB data every few days and I need to pass it
> along enterprise network to the storage center attached to my HPC system, I am
> thinking about compressing it (most tiff format image data) as much as I can, as
> fast as I can before I send it crossing network ... So, I am wondering whether
> anyone is familiar with any hardware based accelerator, which can dramatically
> improve the compressing procedure.. suggestion for any file system architecture
> will be appreciated too.. I have couple of contacts from some vendors but not
> sure whether it works as I expected, so if anyone has experience about it and
> want to share, it will be really appreciated !
Hi Jerry:
Sounds like a bunch of sequencers or arrays going, and dumping data
to a file system. I am not sure you can get deterministic compression
run times from a compressor, or even deterministic compression ratios on
random (binary) data. I have heard of some "xml accelerators" in the
past (back when XML was considered a good buzzword) that did on-the-fly
compression.
I guess it boils down to if
T(compression) + T(comppressed transfer) <<
T(uncompressed_transfer)
And the cost benefit analysis would focus upon the cost of
T(compression) as compared to faster networks. That is, if you spent
$1000/node more to get a faster fabric, which dropped your transfer time
to 20%, is this better/more cost effective than spending $10,000 or so
on an accelerate that may get 70% file compression, and double the
overall time?
Obviously the above numbers are made up, but you get the idea.
Will look around. If you want to talk to a group doing FPGA stuff in
other markets, let me know and I can hook you up. Just be aware that
this might not be cost/time effective.
Joe
>
>
>
> Thanks,
>
> Jerry
>
> Jerry Xu PhD
> HPC Scientific Computing Specialist
> Enterprise Research Infrastructure Systems (ERIS)
> Partners Healthcare, Harvard Medical School
> http://www.partners.org
>
> The information transmitted in this electronic communication is intended only
> for the person or entity to whom it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or other
> use of or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received this
> information in error, please contact the Compliance HelpLine at 800-856-1983 and
> properly dispose of this information.
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Thu Oct 2 17:55:02 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Thu, 2 Oct 2008 14:55:02 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E53FCF.3020008@scalableinformatics.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E53FCF.3020008@scalableinformatics.com>
Message-ID: <20081002215502.GA6059@bx9.net>
On Thu, Oct 02, 2008 at 05:40:31PM -0400, Joe Landman wrote:
> I have heard of some "xml accelerators" in the
> past (back when XML was considered a good buzzword) that did on-the-fly
> compression.
Well, given how wordy the tags are, simply compressing those is
inexpensive and is a big win, if they're a large % of the data.
To get back to the question that was asked, (1) no hardare compresser
compresses smaller than a good software compresser (hardware
compressors tend to be faster but can't compress as well), and (2)
sounds like your data can be compressed in an embarrassingly parallel
fashion, so on a quad-core box you might find that you can keep up
with your link with software compression in parallel.
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From niftyompi at niftyegg.com Thu Oct 2 18:07:16 2008
From: niftyompi at niftyegg.com (Nifty niftyompi Mitch)
Date: Thu, 2 Oct 2008 15:07:16 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID: <20081002220716.GB6110@compegg.wr.niftyegg.com>
On Thu, Oct 02, 2008 at 04:09:36PM -0400, Xu, Jerry wrote:
>
> Currently I generate nearly one TB data every few days and I need to pass it
> along enterprise network to the storage center attached to my HPC system, I am
> thinking about compressing it (most tiff format image data) as much as I can, as
> fast as I can before I send it crossing network ... So, I am wondering whether
> anyone is familiar with any hardware based accelerator, which can dramatically
> improve the compressing procedure.. suggestion for any file system architecture
> will be appreciated too.. I have couple of contacts from some vendors but not
> sure whether it works as I expected, so if anyone has experience about it and
> want to share, it will be really appreciated !
If I recall correctly TIFF files are hard to compress any more than they already are.
Linux has a handful of compression tools -- how much compression are
you able to get on your data with each of these tools and each command line set
of options.
My guess is that the best and most cost effective hardware solution
you will find is a hot Opteron or Intel box with not too many cores and
a good chunk fast DRAM in it.
You might find that contrary to the rest of linux a good optimizing
compiler like PGI, Pathscale, Intel... will speed up the compression code
enough to matter so consider rebuilding things like bzip2, gzip, p7zip
and benchmarking the best compression tool for speed and correctness.
And when you find the best compression program for your data you might
look for some good DSP cards to run your compression on. My bet is that
a new hot box will win. Since this is a cluster mailing list just bang
the compression out to as a cluster job.
--
T o m M i t c h e l l
Found me a new hat, now what?
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From reuti at staff.uni-marburg.de Thu Oct 2 18:09:39 2008
From: reuti at staff.uni-marburg.de (Reuti)
Date: Fri, 3 Oct 2008 00:09:39 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID: <84083434-B04C-4081-8BD0-3F288403F2F7@staff.uni-marburg.de>
Hi,
Am 02.10.2008 um 22:09 schrieb Xu, Jerry:
> Currently I generate nearly one TB data every few days and I need
> to pass it
> along enterprise network to the storage center attached to my HPC
> system, I am
> thinking about compressing it (most tiff format image data)
is it plain tiff or already using any compression like RLE or LZW
inside? Do you want or must stay with tiff?
-- Reuti
> as much as I can, as
> fast as I can before I send it crossing network ... So, I am
> wondering whether
> anyone is familiar with any hardware based accelerator, which can
> dramatically
> improve the compressing procedure.. suggestion for any file system
> architecture
> will be appreciated too.. I have couple of contacts from some
> vendors but not
> sure whether it works as I expected, so if anyone has experience
> about it and
> want to share, it will be really appreciated !
>
>
>
> Thanks,
>
> Jerry
>
> Jerry Xu PhD
> HPC Scientific Computing Specialist
> Enterprise Research Infrastructure Systems (ERIS)
> Partners Healthcare, Harvard Medical School
> http://www.partners.org
>
> The information transmitted in this electronic communication is
> intended only
> for the person or entity to whom it is addressed and may contain
> confidential
> and/or privileged material. Any review, retransmission,
> dissemination or other
> use of or taking of any action in reliance upon this information by
> persons or
> entities other than the intended recipient is prohibited. If you
> received this
> information in error, please contact the Compliance HelpLine at
> 800-856-1983 and
> properly dispose of this information.
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From gdjacobs at gmail.com Thu Oct 2 18:26:39 2008
From: gdjacobs at gmail.com (Geoff Jacobs)
Date: Thu, 02 Oct 2008 17:26:39 -0500
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com>
References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48E4C509.3000407@aplpi.com>
<48E51632.8090001@gmail.com>
<9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com>
Message-ID: <48E54A9F.6020002@gmail.com>
John Hearns wrote:
> Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast.
>
> The FAQ on the site says its a Slackware derivative, if I'm not wrong.
> What goes around comes around I guess :-) Maybe those kipper ties from
> the 70s will be back too.
First question on the page...
http://puppylinux.org/wiki/archives/old-wikka-wikki/categorydocumentation/historypuppy
> Actually, and here I toss in a handgrenade, if we are considering a Damn
> Small Puppy PXE bootable Linux in RAM, what difference does the distro
> make, so long as the GLIBC/maths libraries/MPI libraries are the
> appropriate ones.
> I guess that statement goes for any Linux install really.
Well, you want to use something similar to the head in the compute
nodes, and you want the head node install to have all the necessary
infrastructure. Technically, you could use a different install on each
the nodes, but it wouldn't be smart.
Building a pill for the compute nodes a la Scyld or Perseus seems like
the best compromise in terms of reducing the compute node operating
environment while maintaining a common software base.
--
Geoffrey D. Jacobs
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From grumiche at integrityit.com.br Thu Oct 2 18:37:19 2008
From: grumiche at integrityit.com.br (Rodrigo Grumiche Silva)
Date: Thu, 2 Oct 2008 19:37:19 -0300
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID:
Hi Jerry
I think HDF5 can help you in some way... - http://www.hdfgroup.org/HDF5/
Rodrigo
2008/10/2 Xu, Jerry
> Hello,
>
> Currently I generate nearly one TB data every few days and I need to pass
> it
> along enterprise network to the storage center attached to my HPC system, I
> am
> thinking about compressing it (most tiff format image data) as much as I
> can, as
> fast as I can before I send it crossing network ... So, I am wondering
> whether
> anyone is familiar with any hardware based accelerator, which can
> dramatically
> improve the compressing procedure.. suggestion for any file system
> architecture
> will be appreciated too.. I have couple of contacts from some vendors but
> not
> sure whether it works as I expected, so if anyone has experience about it
> and
> want to share, it will be really appreciated !
>
>
>
> Thanks,
>
> Jerry
>
> Jerry Xu PhD
> HPC Scientific Computing Specialist
> Enterprise Research Infrastructure Systems (ERIS)
> Partners Healthcare, Harvard Medical School
> http://www.partners.org
>
> The information transmitted in this electronic communication is intended
> only
> for the person or entity to whom it is addressed and may contain
> confidential
> and/or privileged material. Any review, retransmission, dissemination or
> other
> use of or taking of any action in reliance upon this information by persons
> or
> entities other than the intended recipient is prohibited. If you received
> this
> information in error, please contact the Compliance HelpLine at
> 800-856-1983 and
> properly dispose of this information.
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From bill at cse.ucdavis.edu Thu Oct 2 21:11:10 2008
From: bill at cse.ucdavis.edu (Bill Broadley)
Date: Thu, 02 Oct 2008 18:11:10 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID: <48E5712E.7070504@cse.ucdavis.edu>
Xu, Jerry wrote:
> Hello,
>
> Currently I generate nearly one TB data every few days and I need to pass it
> along enterprise network to the storage center attached to my HPC system, I am
> thinking about compressing it (most tiff format image data)
tiff uncompressed, or tiff compressed files? If uncompressed I'd guess that
bzip2 might do well with them.
> as much as I can, as
> fast as I can before I send it crossing network ... So, I am wondering whether
> anyone is familiar with any hardware based accelerator, which can dramatically
> improve the compressing procedure..
Improve? You mean compression ratio? Wall clock time? CPU utilization?
Adding forward error correction?
> suggestion for any file system architecture
> will be appreciated too..
Er, hard to imagine a reasonable recommendation without much more information.
Organization, databases (if needed), filenames and related metadata are rather
specific to the circumstances. Access patterns, retention time, backups, and
many other issues would need consideration.
> I have couple of contacts from some vendors but not
> sure whether it works as I expected, so if anyone has experience about it and
> want to share, it will be really appreciated !
Why hardware? I have some python code that managed 10MB/sec per CPU (or 80MB
on 8 CPUs if you prefer) that compresses with zlib, hashes with sha256, and
encrypts with AES (256 bit key). Assuming the compression you want isn't
substantially harder than doing zlib, sha256, and aes a single core from a
dual or quad core chip sold in the last few years should do fine.
1TB every 2 days = 6MB/sec or approximately 15% of a quad core or 60% of a
single core for my compress, hash and encrypt in python. Considering how
cheap cores are (quad desktops are often under $1k) I'm not sure what would
justify an accelerator card. Not to mention picking the particular algorithm
could make a huge difference to the CPU and compression ratio achieved. I'd
recommend taking a stack of real data and trying out different compression
tools and settings.
In any case 6MB/sec of compression isn't particularly hard these days.... even
in python on a 1-2 year old mid range cpu.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hahn at mcmaster.ca Thu Oct 2 22:31:48 2008
From: hahn at mcmaster.ca (Mark Hahn)
Date: Thu, 2 Oct 2008 22:31:48 -0400 (EDT)
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID:
> Currently I generate nearly one TB data every few days and I need to pass it
Bill's right - 6 MB/s is really not much to ask from even a complex WAN.
I think the first thing you should do is find the bottleneck. to me it
sounds like you have a sort of ropey path with a 100 Mbps hop somewhere.
> thinking about compressing it (most tiff format image data) as much as I can
tiff is a fairly generic container that can hold anything from a horrible
uncompressed 4-byte-per-pixel to jpeg or rle. looking at the format you're
really using would be wise. I'm guessing that if you transcode to png,
you'll get better compression than gzip/etc. dictionary-based compression
is fundamentally inappropriate for most non-text data - not images, not
double-precision dumps of physical simulations, etc. png is quite a lot
smarter about most kinds of images than older formats, and can be lossy or
lossless.
hardware compression would be a serious mistake unless you've already
pursued these routes. specialized hardware is a very short-term and
quite narrow value proposition. I would always prefer to improve the
infrastructure.
> The information transmitted in this electronic communication is intended only
uh, email is publication.
regards, mark hahn.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From coutinho at dcc.ufmg.br Thu Oct 2 22:42:32 2008
From: coutinho at dcc.ufmg.br (Bruno Coutinho)
Date: Thu, 2 Oct 2008 23:42:32 -0300
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5712E.7070504@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
Message-ID:
2008/10/2 Bill Broadley
Why hardware? I have some python code that managed 10MB/sec per CPU (or
> 80MB
> on 8 CPUs if you prefer) that compresses with zlib, hashes with sha256, and
> encrypts with AES (256 bit key). Assuming the compression you want isn't
> substantially harder than doing zlib, sha256, and aes a single core from a
> dual or quad core chip sold in the last few years should do fine.
>
> 1TB every 2 days = 6MB/sec or approximately 15% of a quad core or 60% of a
> single core for my compress, hash and encrypt in python. Considering how
> cheap cores are (quad desktops are often under $1k) I'm not sure what would
> justify an accelerator card. Not to mention picking the particular
> algorithm
> could make a huge difference to the CPU and compression ratio achieved.
> I'd
> recommend taking a stack of real data and trying out different compression
> tools and settings.
>
> In any case 6MB/sec of compression isn't particularly hard these days....
> even
> in python on a 1-2 year old mid range cpu.
>
>
>
In Information Retrieval, they compress almost everything and they have
papers showing that using compression can result in a *faster* system. You
process a little more, but get great gains in disk throughput.
If you compress before even storing data, your system could store faster by
using less disk/storage bandwidth per stored file.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 04:13:16 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 10:13:16 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5712E.7070504@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
Message-ID: <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
Bzip2, gzip,
Why do you guys keep quoting those total outdated compressors :)
there is 7-zip for linux, it's open source and also part of LZMA. On
average remnants
are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/gzip
is factor 2 worse).
7-zip also works parallel, not sure whether it works in linux
parallel. 7za is command line
version.
Linux distributions should include it default.
Uses PPM, that's a new form of multidimensional compression that all
that old junk like
bzip2/gzip doesn't use.
TIFF files compress real bad of course. Maybe convert them to some
more inefficient format,
which increases its size probably, which then compresses real great
with PPM.
When googling for the best compressors, don't try PAQ, that's a
benchmark compressor. Was worse for my
terabyte of data than even 7-zip (which is not by far best PPM
compressor, but it's open source).
Vincent
On Oct 3, 2008, at 3:11 AM, Bill Broadley wrote:
> Xu, Jerry wrote:
>> Hello, Currently I generate nearly one TB data every few days and
>> I need to pass it
>> along enterprise network to the storage center attached to my HPC
>> system, I am
>> thinking about compressing it (most tiff format image data)
>
> tiff uncompressed, or tiff compressed files? If uncompressed I'd
> guess that
> bzip2 might do well with them.
>
>> as much as I can, as
>> fast as I can before I send it crossing network ... So, I am
>> wondering whether
>> anyone is familiar with any hardware based accelerator, which can
>> dramatically
>> improve the compressing procedure..
>
> Improve? You mean compression ratio? Wall clock time? CPU
> utilization?
> Adding forward error correction?
>
>> suggestion for any file system architecture
>> will be appreciated too..
>
> Er, hard to imagine a reasonable recommendation without much more
> information.
> Organization, databases (if needed), filenames and related metadata
> are rather
> specific to the circumstances. Access patterns, retention time,
> backups, and many other issues would need consideration.
>
>> I have couple of contacts from some vendors but not
>> sure whether it works as I expected, so if anyone has experience
>> about it and
>> want to share, it will be really appreciated !
>
> Why hardware? I have some python code that managed 10MB/sec per
> CPU (or 80MB
> on 8 CPUs if you prefer) that compresses with zlib, hashes with
> sha256, and
> encrypts with AES (256 bit key). Assuming the compression you want
> isn't
> substantially harder than doing zlib, sha256, and aes a single core
> from a
> dual or quad core chip sold in the last few years should do fine.
>
> 1TB every 2 days = 6MB/sec or approximately 15% of a quad core or
> 60% of a
> single core for my compress, hash and encrypt in python.
> Considering how
> cheap cores are (quad desktops are often under $1k) I'm not sure
> what would
> justify an accelerator card. Not to mention picking the particular
> algorithm
> could make a huge difference to the CPU and compression ratio
> achieved. I'd
> recommend taking a stack of real data and trying out different
> compression
> tools and settings.
>
> In any case 6MB/sec of compression isn't particularly hard these
> days.... even
> in python on a 1-2 year old mid range cpu.
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From bill at cse.ucdavis.edu Fri Oct 3 05:17:52 2008
From: bill at cse.ucdavis.edu (Bill Broadley)
Date: Fri, 03 Oct 2008 02:17:52 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
Message-ID: <48E5E340.6080408@cse.ucdavis.edu>
Vincent Diepeveen wrote:
> Bzip2, gzip,
>
> Why do you guys keep quoting those total outdated compressors :)
Path of least resistance, not to mention python bindings.
> there is 7-zip for linux, it's open source and also part of LZMA. On
> average remnants
> are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/gzip is
> factor 2 worse).
> 7-zip also works parallel, not sure whether it works in linux parallel.
> 7za is command line
> version.
Seems like the question is related to CPU utilization as well as compression
ratios. Assuming the TIFF files are not already compressed, how fast would
you expect 7-zip to be relative to bzip2 and gzip's compression and
decompression speeds? I was looking for decent bandwidth, and I did look
around a bit and it seemed like things often would compress somewhat better,
often the bandwidth achieved was 5-6x worse. So for squeezing the most out of
a 28k modem... sure. For keeping up with a 100mbit or GigE connection on a
local LAN, not so much.
Google finds:
http://blogs.reucon.com/srt/2008/02/18/compression_gzip_vs_bzip2_vs_7_zip.html
Compressor Size Ratio Compression Decompression
gzip 89 MB 54 % 0m 13s 0m 05s
bzip2 81 MB 49 % 1m 30s 0m 20s
7-zip 61 MB 37 % 1m 48s 0m 11s
So sure you save 28MB, at the cost of 95 seconds. Might make sense if you are
transfering over a slow modem. Also considering the original file was 163MB
it's nowhere near the 6MB/sec that seems to be the target. At 1.5MB/sec you'd
need 4 CPUs running flat out for 2 days to manage 2TB, instead of 1 CPU
running for just 24 hours. Definitely the kind of thing that sounds like it
might make a big difference.
Another example:
http://bbs.archlinux.org/viewtopic.php?t=11670
7zip compress: 19:41
Bzip2 compress: 8:56
Gzip compress: 3:00
Again 7zip is a factor of 6 and change slower than gzip.
> Linux distributions should include it default.
>
> Uses PPM, that's a new form of multidimensional compression that all
> that old junk like
> bzip2/gzip doesn't use.
One man's junk and another man's gold. My use was backup related and I
definitely didn't want to become CPU limited even on large systems with 10TB
of disk and a healthy I/O system. From the sounds of it even with 8 fast
cores that 7zip might easily be the bottleneck.
> TIFF files compress real bad of course. Maybe convert them to some more
> inefficient format,
> which increases its size probably, which then compresses real great with
> PPM.
Er, that makes no sense to me. You aren't going to end up with a smaller file
by encoding a file less efficiently.. under ideal circumstances you might get
back to where you started with a substantial use of cycles. Seems pretty
simple, if the TIFFs are compressed, just send them as is, significant
additional compression is unlikely. If they are uncompressed there's a decent
chance of significant lossless compression, the best thing to do would be to
try it or at least a reference to some similar images.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From carsten.aulbert at aei.mpg.de Fri Oct 3 05:27:36 2008
From: carsten.aulbert at aei.mpg.de (Carsten Aulbert)
Date: Fri, 03 Oct 2008 11:27:36 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu>
Message-ID: <48E5E588.1050508@aei.mpg.de>
Hi all
Bill Broadley wrote:
>
> Another example:
> http://bbs.archlinux.org/viewtopic.php?t=11670
>
> 7zip compress: 19:41
> Bzip2 compress: 8:56
> Gzip compress: 3:00
>
> Again 7zip is a factor of 6 and change slower than gzip.
Have you looked into threaded/parallel bzip2?
freshmeat has a few of those, e.g.
http://freshmeat.net/projects/bzip2smp/
http://freshmeat.net/projects/lbzip2/
(with the usual disclaimer that I haven't tested them myself).
HTH
carsten
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From bill at cse.ucdavis.edu Fri Oct 3 05:28:52 2008
From: bill at cse.ucdavis.edu (Bill Broadley)
Date: Fri, 03 Oct 2008 02:28:52 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu>
Message-ID: <48E5E5D4.5030102@cse.ucdavis.edu>
For uncompressed TIFF files this might be of use:
http://optipng.sourceforge.net/features.html
It seems to mention the lossless compression of TIFF files.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 06:29:16 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 12:29:16 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu>
Message-ID:
hi Bill,
7-zip is of course faster than bzip2 and a bit slower than gzip.
Thing is that after the files have been compressed you need to do
less i/o;
the big bottleneck in most systems is the bandwidth from and to i/o.
A single disk start of 90s was several megabytes per second up to 10
MB/s.
Speed i see SATA2 drives get now is roughly 133MB/s.
I remember a speech from NCSA director a few weeks ago where he
showed a graph,
and it's quite obvious that we are gonna get enough cpu power in future,
what to do with it?
Let's be realistic. Suppose you've got a bunch of cores. Soon hundreds.
Then for each byte you get off disk, you have hundreds of cpu cycles
to decompress.
That bandwidth to and from i/o you can limit bigtime with a bigger
compression,
cpu power there is enough, provided it's not taking too long to
decompress.
Decompression speed of 7-zip is faster than bzip2 in my experience,
simply
because it can decompress in parallel and i don't see bzip2 ever
getting parallellized.
Compression is one of those areas where there gets put little money
in making a real good
realtime compression. In fact i'm busy making a kind of filesystem
myself now where compression
is allowed to take long and decompression must be realtime (simply at
a given cpu speed a certain
bandwidth to decompress) for any bucket.
Best compressors today have just 1 basic problem for GIS type and in
my case chess database
systems; they decompress the entire file.
With files of gigabytes, if you just want to access random byte
number X in the file, you're not interested
in bandwidth nor whether gzip is faster or slower than PPM type
compressors in decompression.
You just want that single byte that might be at the end of the file.
I'm planning to build such a compressor which is doing a lot of
mathematical function comparisions to
get PPM type compression, yet taking care decompression can be done
realtime.
there is a few filesystems which allow this, but the compression they
use for it, is just 70s/80s huffman based.
Real ugly bad, to say polite.
Note i'm speaking of lossless compression here, lossy compression is
yet another subject.
On the web if you ask me, too many photo's are of a quality that is
just ugly. Lots to improve there as well.
Lossy you can do way more of course than lossless.
Yet lossless i do not see real great systems there yet.
Time to build one myself :)
My argument is that something like a GPU might be real interesting to
take a look at for compression.
You can dedicate a bunch of cheapo GPU's to just compression; for
databases what really matters is the
decompression time.
They store so much, i'd argue a good compression, both lossless and
lossy is what we need;
Nowadays governments want to put in database every phoneconversation,
no matter how hard they're gonna deny that
to you;
I'd forsee that the amount of storage needed is huge, tiny company i
worked for was putting away in petabyte level,
and majority will never get decompressed, other than for some AI type
software programs that do some scanning later on.
The real big difference between the 'testsets' that get used in
compression and the status of compression, is that there is no
money to make for those guys writing a superior compressor that works
in terabytes. Companies do not soon invest in such
technologies, they just invest in products they can sell.
Having also worked with GIS data i see possibilities to compress that
factors better and still realtime look it up from embedded cpu's.
Vincent
On Oct 3, 2008, at 11:17 AM, Bill Broadley wrote:
> Vincent Diepeveen wrote:
>> Bzip2, gzip,
>> Why do you guys keep quoting those total outdated compressors :)
>
> Path of least resistance, not to mention python bindings.
>
>> there is 7-zip for linux, it's open source and also part of LZMA.
>> On average remnants
>> are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/
>> gzip is factor 2 worse).
>> 7-zip also works parallel, not sure whether it works in linux
>> parallel. 7za is command line
>> version.
>
> Seems like the question is related to CPU utilization as well as
> compression ratios. Assuming the TIFF files are not already
> compressed, how fast would you expect 7-zip to be relative to bzip2
> and gzip's compression and decompression speeds? I was looking for
> decent bandwidth, and I did look around a bit and it seemed like
> things often would compress somewhat better, often the bandwidth
> achieved was 5-6x worse. So for squeezing the most out of a 28k
> modem... sure. For keeping up with a 100mbit or GigE connection on
> a local LAN, not so much.
>
> Google finds:
> http://blogs.reucon.com/srt/2008/02/18/
> compression_gzip_vs_bzip2_vs_7_zip.html
>
> Compressor Size Ratio Compression Decompression
> gzip 89 MB 54 % 0m 13s 0m 05s
> bzip2 81 MB 49 % 1m 30s 0m 20s
> 7-zip 61 MB 37 % 1m 48s 0m 11s
>
> So sure you save 28MB, at the cost of 95 seconds. Might make sense
> if you are transfering over a slow modem. Also considering the
> original file was 163MB it's nowhere near the 6MB/sec that seems to
> be the target. At 1.5MB/sec you'd need 4 CPUs running flat out for
> 2 days to manage 2TB, instead of 1 CPU running for just 24 hours.
> Definitely the kind of thing that sounds like it might make a big
> difference.
>
> Another example:
> http://bbs.archlinux.org/viewtopic.php?t=11670
>
> 7zip compress: 19:41
> Bzip2 compress: 8:56
> Gzip compress: 3:00
>
> Again 7zip is a factor of 6 and change slower than gzip.
>
>> Linux distributions should include it default.
>> Uses PPM, that's a new form of multidimensional compression that
>> all that old junk like
>> bzip2/gzip doesn't use.
>
> One man's junk and another man's gold. My use was backup related
> and I definitely didn't want to become CPU limited even on large
> systems with 10TB of disk and a healthy I/O system. From the
> sounds of it even with 8 fast cores that 7zip might easily be the
> bottleneck.
>
>> TIFF files compress real bad of course. Maybe convert them to some
>> more inefficient format,
>> which increases its size probably, which then compresses real
>> great with PPM.
>
> Er, that makes no sense to me. You aren't going to end up with a
> smaller file by encoding a file less efficiently.. under ideal
> circumstances you might get back to where you started with a
> substantial use of cycles. Seems pretty simple, if the TIFFs are
> compressed, just send them as is, significant additional
> compression is unlikely. If they are uncompressed there's a decent
> chance of significant lossless compression, the best thing to do
> would be to try it or at least a reference to some similar images.
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 06:53:49 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 12:53:49 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E588.1050508@aei.mpg.de>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de>
Message-ID: <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
hi Carsten,
Ah you googled 2 seconds and found some oldie homepage.
Try this homepage www.maximumcompression.com
Far better testing over there. Note that it's the same testset there
that gets compressed a lot.
In real life, database type data is having all kind of patterns which
PPM type compressors find.
My experience is that at terabyte level the better compressors at
maximumcompression.com,
are a bit too slow (PAQ) and not so good like simple things like 7-zip.
Look especially at compressed sizes and decompression times.
The only thing you want to limit over your network is the amount of
bandwidth over your network.
A real good compression is very helpful then. How long compression
time takes is nearly not relevant,
as long as it doesn't take infinite amounts of time (i remember a new
zealand compressor which took 24
hours to compress a 100MB data). Note that we are already at a phase
that compression time hardly
matters, you can buy a GPU for that to offload your servers for that.
Query time (so decompression time) is important though.
If we look to graphics there:
026 7-Zip 4.60b -m0=ppmd:o=4 764420 81.58 1.4738
..
94 BZIP2 1.0.5 -9 890163 78.55
1.7162
..
158 PKZIP 2.50 -exx 1250536 69.86
2.4110
159 HIT 2.10 -x 1250601 69.86
2.4111
160 GZIP 1.3.5 -9 1254351 69.77 2.4184
161 ZIP 2.2 -9 1254444 69.77
2.4185
162 WINZIP 8.0 (Max Compression) 1254444 69.77 2.4185
Note a real supercompressor is getting it even tinier:
003 WinRK 3.0.3 PWCM 912MB 568919 86.29 1.0969
Again all these tests are at microlevel. Just a few megabtes of data
that gets compressed.
You don't build a big infrastructure just for a few megabytes, it's
not so relevant.
The traffic over your network dominates there, plenty of idle server
cores there is, in fact there is
so many companies now that buy dual cores, as they do not know how to
keep the cores in quad cores
busy.
This is all microlevel. Things really change when you have terabytes
to compress and HUGE files.
Bzip2 is ugly slow for files in gigabyte size, 7-zip is totally
beating it there.
Vincent
On Oct 3, 2008, at 11:27 AM, Carsten Aulbert wrote:
> Hi all
>
> Bill Broadley wrote:
>>
>> Another example:
>> http://bbs.archlinux.org/viewtopic.php?t=11670
>>
>> 7zip compress: 19:41
>> Bzip2 compress: 8:56
>> Gzip compress: 3:00
>>
>> Again 7zip is a factor of 6 and change slower than gzip.
>
> Have you looked into threaded/parallel bzip2?
>
> freshmeat has a few of those, e.g.
>
> http://freshmeat.net/projects/bzip2smp/
> http://freshmeat.net/projects/lbzip2/
>
> (with the usual disclaimer that I haven't tested them myself).
>
> HTH
>
> carsten
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hahn at mcmaster.ca Fri Oct 3 10:00:05 2008
From: hahn at mcmaster.ca (Mark Hahn)
Date: Fri, 3 Oct 2008 10:00:05 -0400 (EDT)
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To:
References: <48E39782.4020006@ias.edu>
Message-ID:
>> Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
>
> I don't think front side busses are fast enough for QDR yet.
qdr would be 40 Gb (raw data rate, right? so 4 GB/s before any sort
of packet overhead, etc.) I don't really see why that's a problem -
even a memory-constrained current-gen Intel box has about twice
that much memory bandwidth available. AMD or next-gen-Intel will
be even less constrained.
otoh, I rarely see workloads that are clearly net-bw-limited even
on 4-core myrinet 2g machines - and that's only about 250 MB/s.
regards, mark hahn.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From carsten.aulbert at aei.mpg.de Fri Oct 3 07:49:04 2008
From: carsten.aulbert at aei.mpg.de (Carsten Aulbert)
Date: Fri, 03 Oct 2008 13:49:04 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de>
<5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
Message-ID: <48E606B0.3050708@aei.mpg.de>
Hi Vincent,
Vincent Diepeveen wrote:
> Ah you googled 2 seconds and found some oldie homepage.
Actually no, I just looked at my freshmeat watchlist of items still to
look at :)
>
> Look especially at compressed sizes and decompression times.
Yeah, I'm currently looking at
http://www.maximumcompression.com/data/summary_mf3.php
We have a Gbit network, i.e. for us this test is a null test, since it
takes 7-zip close to 5 minutes to compress the data set of 311 MB which
we could blow over the network in less than 5 seconds, i.e. in this case
tar would be our favorite ;)
>
> The only thing you want to limit over your network is the amount of
> bandwidth over your network.
> A real good compression is very helpful then. How long compression time
> takes is nearly not relevant,
> as long as it doesn't take infinite amounts of time (i remember a new
> zealand compressor which took 24
> hours to compress a 100MB data). Note that we are already at a phase
> that compression time hardly
> matters, you can buy a GPU for that to offload your servers for that.
>
No, quite on the contrary. I would like to use a compressor within a
pipe to increase the throughput over the network, i.e. to get around the
~ 120 MB/s limit.
> Query time (so decompression time) is important though.
>
No for us that number is at least as important than the compression time.
Imagine the following situation:
We have a file server with close to 10 TB of data on it in nice chunks
with a since of about 100 MB each. We buy a new server with new disks
and the new one can now hold 20 TB and we would like to copy the data
over. So for us the more important figure is the
compression/decompression speed which should be >> 100 MB/s on our systems.
If 7-zip can only compress data at a rate of less than say 5 MB/s (input
data) I can much much faster copy the data over uncompressed regardless
of how many unused cores I have in the system. Exactly for these cases I
would like to use all cores available to compress the data fast in order
to increase the throughput.
Do I miss something vital?
Cheers
Carsten
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 11:27:36 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 17:27:36 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E606B0.3050708@aei.mpg.de>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de>
<5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
Message-ID:
Hi Carsten,
In your example the only thing that seems to matter to you is
*collecting* data speed,
in short the realtime compression speed that tapestreamers can get,
to give one
example.
In your example you need to compress each time stuff.
That's not being realistic however. I'll give you 2 arguments.
In an ideal situation you compress things only a single time. So
decompression speed
matters if you want to realtime lookup data. It would be far more
ideal in your situation to
already have things very well compressed at your drive, doing some
realtime compression/decompression
is not very useful then, the hardware compression of those tape
streamers already is doing some simplistico
Runlength encoding usually.
Now another lack is that you assume stuff that's on your 10TB array
is never getting used by not a single user
over the network.
Vincent
On Oct 3, 2008, at 1:49 PM, Carsten Aulbert wrote:
> Hi Vincent,
>
> Vincent Diepeveen wrote:
>> Ah you googled 2 seconds and found some oldie homepage.
>
> Actually no, I just looked at my freshmeat watchlist of items still to
> look at :)
>
>>
>> Look especially at compressed sizes and decompression times.
>
> Yeah, I'm currently looking at
> http://www.maximumcompression.com/data/summary_mf3.php
>
> We have a Gbit network, i.e. for us this test is a null test, since it
> takes 7-zip close to 5 minutes to compress the data set of 311 MB
> which
> we could blow over the network in less than 5 seconds, i.e. in this
> case
> tar would be our favorite ;)
>
>>
>> The only thing you want to limit over your network is the amount of
>> bandwidth over your network.
>> A real good compression is very helpful then. How long compression
>> time
>> takes is nearly not relevant,
>> as long as it doesn't take infinite amounts of time (i remember a new
>> zealand compressor which took 24
>> hours to compress a 100MB data). Note that we are already at a phase
>> that compression time hardly
>> matters, you can buy a GPU for that to offload your servers for that.
>>
>
> No, quite on the contrary. I would like to use a compressor within a
> pipe to increase the throughput over the network, i.e. to get
> around the
> ~ 120 MB/s limit.
>
>> Query time (so decompression time) is important though.
>>
>
> No for us that number is at least as important than the compression
> time.
>
> Imagine the following situation:
>
> We have a file server with close to 10 TB of data on it in nice chunks
> with a since of about 100 MB each. We buy a new server with new disks
> and the new one can now hold 20 TB and we would like to copy the data
> over. So for us the more important figure is the
> compression/decompression speed which should be >> 100 MB/s on our
> systems.
>
> If 7-zip can only compress data at a rate of less than say 5 MB/s
> (input
> data) I can much much faster copy the data over uncompressed
> regardless
> of how many unused cores I have in the system. Exactly for these
> cases I
> would like to use all cores available to compress the data fast in
> order
> to increase the throughput.
>
> Do I miss something vital?
>
> Cheers
>
> Carsten
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Fri Oct 3 11:45:43 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Fri, 03 Oct 2008 11:45:43 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E606B0.3050708@aei.mpg.de>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
Message-ID: <48E63E27.6060701@scalableinformatics.com>
Carsten Aulbert wrote:
> If 7-zip can only compress data at a rate of less than say 5 MB/s (input
> data) I can much much faster copy the data over uncompressed regardless
> of how many unused cores I have in the system. Exactly for these cases I
> would like to use all cores available to compress the data fast in order
> to increase the throughput.
This is fundamentally the issue. If the compression time plus the
tranmit time for the compressed data is greater than the transmit time
for the uncompressed data, then the compression may not be worth it.
Sure, if it is nothing but text files, you may get 60-80+% compression
rates. But for the case of (non-pathological) binary data, it might be
only a few percent. So in this case, even if you could get a few
percent delta from the compression, is that worth all the extra time you
spend to get it?
At the end of the day the question is how much lossless compression can
you do in a short enough time for it to be meaningful in terms of
transmitting the data?
>
> Do I miss something vital?
Nope. You got it nailed.
Several months ago, I tried moving about 600 GB of data from an old
server to a JackRabbit. The old server and the JackRabbit had a gigabit
link between them. We regularly saw 45 MB scp rates (one of the chips
in the older server was a Broadcom).
I tried this with and without compression. With compression (simple
gzip), the copy took something like 28 hours ( a little more than a
day). Without compression, it was well under 10 hours.
In this case, compression (gzip) was not worth it. The command I used
for the test was
uncompressed:
cd /directory
tar -cpf - ./ | ssh jackrabbit "cd /directory ; tar -xpvf - "
compressed:
cd /directory
tar -czpf - ./ | ssh jackrabbit "cd /directory ; tar -xzpvf - "
if you want to spend more time, use "j" rather than "z" in the options.
YMMV, but I have been convinced that, apart from specific use cases with
text only documents or documents known to compress quickly/well, that
compression prior to transfer may waste more time than it saves.
This said, if someone has a parallel hack of gzip or similar we can pipe
through, by all means, I would be happy to try it. But it would have to
be pretty darned efficient.
100MB/s means 1 byte transmitted,on average, in 10 nanoseconds. Which
means for compression to be meaningful, you would need to compute for
less time than that to increase the information density. Put another
way, 1 MB takes about 10 ms to send over a gigabit link. For
compression to be meaningful, you need to compress this 1MB in far less
than 10 ms and still transmit it in that time. Assuming that any
compression algorithm has to walk through data at least once, A 1 GB/s
memory subsystem takes about 1 ms to walk through this data at least
once, so you need as few passes as possible through the data set to
construct the compressed representation, as you will still have on the
order of 1E+5 bytes to send.
I am not saying it is hopeless, just hard for complex compression
schemes (bzip2, etc). When we get enough firepower in the CPU (or maybe
GPU ... hmmmm) the situation may improve.
GPU as a compression engine? Interesting ...
Joe
>
> Cheers
>
> Carsten
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 11:55:08 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 17:55:08 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E63E27.6060701@scalableinformatics.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
<48E63E27.6060701@scalableinformatics.com>
Message-ID: <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl>
The question is Joe,
Why are you storing it uncompressed?
Vincent
On Oct 3, 2008, at 5:45 PM, Joe Landman wrote:
> Carsten Aulbert wrote:
>
>> If 7-zip can only compress data at a rate of less than say 5 MB/s
>> (input
>> data) I can much much faster copy the data over uncompressed
>> regardless
>> of how many unused cores I have in the system. Exactly for these
>> cases I
>> would like to use all cores available to compress the data fast in
>> order
>> to increase the throughput.
>
> This is fundamentally the issue. If the compression time plus the
> tranmit time for the compressed data is greater than the transmit
> time for the uncompressed data, then the compression may not be
> worth it. Sure, if it is nothing but text files, you may get 60-80+
> % compression rates. But for the case of (non-pathological) binary
> data, it might be only a few percent. So in this case, even if
> you could get a few percent delta from the compression, is that
> worth all the extra time you spend to get it?
>
> At the end of the day the question is how much lossless compression
> can you do in a short enough time for it to be meaningful in terms
> of transmitting the data?
>
>> Do I miss something vital?
>
> Nope. You got it nailed.
>
> Several months ago, I tried moving about 600 GB of data from an old
> server to a JackRabbit. The old server and the JackRabbit had a
> gigabit link between them. We regularly saw 45 MB scp rates (one
> of the chips in the older server was a Broadcom).
>
> I tried this with and without compression. With compression
> (simple gzip), the copy took something like 28 hours ( a little
> more than a day). Without compression, it was well under 10 hours.
>
> In this case, compression (gzip) was not worth it. The command I
> used for the test was
>
> uncompressed:
>
> cd /directory
> tar -cpf - ./ | ssh jackrabbit "cd /directory ; tar -xpvf - "
>
> compressed:
>
> cd /directory
> tar -czpf - ./ | ssh jackrabbit "cd /directory ; tar -xzpvf - "
>
> if you want to spend more time, use "j" rather than "z" in the
> options.
>
> YMMV, but I have been convinced that, apart from specific use cases
> with text only documents or documents known to compress quickly/
> well, that compression prior to transfer may waste more time than
> it saves.
>
> This said, if someone has a parallel hack of gzip or similar we can
> pipe through, by all means, I would be happy to try it. But it
> would have to be pretty darned efficient.
>
> 100MB/s means 1 byte transmitted,on average, in 10 nanoseconds.
> Which means for compression to be meaningful, you would need to
> compute for less time than that to increase the information
> density. Put another way, 1 MB takes about 10 ms to send over a
> gigabit link. For compression to be meaningful, you need to
> compress this 1MB in far less than 10 ms and still transmit it in
> that time. Assuming that any compression algorithm has to walk
> through data at least once, A 1 GB/s memory subsystem takes about
> 1 ms to walk through this data at least once, so you need as few
> passes as possible through the data set to construct the compressed
> representation, as you will still have on the order of 1E+5 bytes
> to send.
>
> I am not saying it is hopeless, just hard for complex compression
> schemes (bzip2, etc). When we get enough firepower in the CPU (or
> maybe GPU ... hmmmm) the situation may improve.
>
> GPU as a compression engine? Interesting ...
>
> Joe
>
>> Cheers
>> Carsten
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web : http://www.scalableinformatics.com
> http://jackrabbit.scalableinformatics.com
> phone: +1 734 786 8423 x121
> fax : +1 866 888 3112
> cell : +1 734 612 4615
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Fri Oct 3 11:59:53 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Fri, 03 Oct 2008 11:59:53 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
<48E63E27.6060701@scalableinformatics.com>
<0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl>
Message-ID: <48E64179.8070808@scalableinformatics.com>
Vincent Diepeveen wrote:
> The question is Joe,
>
> Why are you storing it uncompressed?
Lets see now...
Riddle me this Vincent: What happens when you compress uncompressable
binary data? Like RPMs, or .debs? Or pdf's with images, or images
which have already been compressed?
Any idea?
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 12:00:16 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 18:00:16 +0200
Subject: [Beowulf] GPU (was Accelerator) for data compressing
In-Reply-To: <48E63E27.6060701@scalableinformatics.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
<48E63E27.6060701@scalableinformatics.com>
Message-ID: <6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl>
On Oct 3, 2008, at 5:45 PM, Joe Landman wrote:
>
> GPU as a compression engine? Interesting ...
>
> Joe
>
For great compression, it's rather hard to get that to work.
With a lot of RAM some clever guys manage.
GPU has a lot of stream processors, yet little RAM a stream processor.
Additionally they all have to execute the same code at a bunch of
SP's at the same time.
So there is a big need for some real clever new algorithm there,
as the lack of RAM is gonna hurt really bigtime.
Would be a mighty interesting study to get something to work there.
It allows real
complicated mathematical functions for PPM functionality.
What's in that GPU soon will be in a CPU anyway, so it benefits the
entire planet a thing like that.
Where can i ask for funding?
Vincent
>> Cheers
>> Carsten
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web : http://www.scalableinformatics.com
> http://jackrabbit.scalableinformatics.com
> phone: +1 734 786 8423 x121
> fax : +1 866 888 3112
> cell : +1 734 612 4615
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Fri Oct 3 12:13:46 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Fri, 3 Oct 2008 18:13:46 +0200
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E64179.8070808@scalableinformatics.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
<48E63E27.6060701@scalableinformatics.com>
<0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl>
<48E64179.8070808@scalableinformatics.com>
Message-ID: <83F10C80-F97B-48D2-A586-260DCD4BD9A0@xs4all.nl>
Well that's obvious what happens there, that's not really new,
already the old good pkzip was storing files it couldn't compress
uncompressed.
Note the type of files you mention you can compress quite well still
with PPM, which really is nothing new anymore.
All the old zippers are LZ77/Huffman/RLE based, so they miss the big
picture always.
JPG is a lot tougher to compress well, but well as i said, there is
always the cheap option of a filesystem storing that uncompressesd :)
What gets shipped however most over the networks is XML data from all
those media networks.
Now that compresses *real well*.
Just compress it single time and keep it compressed :)
Vincent
On Oct 3, 2008, at 5:59 PM, Joe Landman wrote:
> Vincent Diepeveen wrote:
>> The question is Joe,
>> Why are you storing it uncompressed?
>
> Lets see now...
>
> Riddle me this Vincent: What happens when you compress
> uncompressable binary data? Like RPMs, or .debs? Or pdf's with
> images, or images which have already been compressed?
>
> Any idea?
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web : http://www.scalableinformatics.com
> http://jackrabbit.scalableinformatics.com
> phone: +1 734 786 8423 x121
> fax : +1 866 888 3112
> cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From mathog at caltech.edu Fri Oct 3 12:55:48 2008
From: mathog at caltech.edu (David Mathog)
Date: Fri, 03 Oct 2008 09:55:48 -0700
Subject: [Beowulf] Re: Accelerator for data compressing
Message-ID:
Carsten Aulbert wrote
> We have a Gbit network, i.e. for us this test is a null test, since it
> takes 7-zip close to 5 minutes to compress the data set of 311 MB which
> we could blow over the network in less than 5 seconds, i.e. in this case
> tar would be our favorite ;)
Many compression programs have a parameter to adjust how hard it tries
to squeeze things, offering a trade off between speed and compression.
For 7za you want to look at the -m switch, see:
http://www.bugaco.com/7zip/MANUAL/switches/method.htm
That looks a little old, on my system this file is:
/usr/share/doc/p7zip/MANUAL/switches/method.htm
Depending on what you are sending, and it all depends on that, you might
get some speed up with a simple run length encoding (for data that tends
to be in long blocks), word or character encoding (for highly repetitive
data, like DNA), or not be able to do any better no matter what
compression method you use (random bits.)
One of these is always rate limiting in data distribution: read,
compression, transmit, receive, decompress, write. Ignoring
compression, which of these hardware parameters is rate limiting
in your case? If it is read or write you can speed things up by
adopting a different storage configuration (details vary, but
underneath it will come down to spooling data off of/onto N disks at
once, instead of just 1, to multiply the disk IO by N.)
Regards,
David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From prentice at ias.edu Fri Oct 3 13:53:01 2008
From: prentice at ias.edu (Prentice Bisbal)
Date: Fri, 03 Oct 2008 13:53:01 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E588.1050508@aei.mpg.de>
References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu>
<48E5E588.1050508@aei.mpg.de>
Message-ID: <48E65BFD.1050607@ias.edu>
Carsten Aulbert wrote:
> > Hi all
> >
> > Bill Broadley wrote:
>> >> Another example:
>> >> http://bbs.archlinux.org/viewtopic.php?t=11670
>> >>
>> >> 7zip compress: 19:41
>> >> Bzip2 compress: 8:56
>> >> Gzip compress: 3:00
>> >>
>> >> Again 7zip is a factor of 6 and change slower than gzip.
> >
> > Have you looked into threaded/parallel bzip2?
> >
If you can parallelize compression, has anyone done any work using a GPU
to do this? Maybe there's a CUDA-fied gzip/bzip2/7sip out there. IF that
was the case, a cheap video card could do the trick, assuming the
machines in question have slots/space for such a video card.
I'm googling now, but not finding anything. :( It was just a thought...
-- Prentice
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From award at uda.ad Fri Oct 3 14:06:02 2008
From: award at uda.ad (Alan Ward)
Date: Fri, 3 Oct 2008 20:06:02 +0200
Subject: [Beowulf] Streamlined standard Linux installation
Message-ID:
Hi.
Several days ago there was a thread on using a streamlined standard Linux distribution. These days I have been fiddling with the new Debian Lenny version (still beta, but it seems not for long).
A "no frills, no X" install with vi, aptitude and a few other rescue tools fits comfortably into about 300 Mbytes on an old USB pendrive. Setting a maximum of 512 MBytes for a basic cluster node setup seems feasable for some applications.
Boot time from GRUB to login prompt is 22s on a 6-year-old Celeron laptop.
And no - no OpenOffice on this one! ;-)
Cheers,
-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From bill at cse.ucdavis.edu Fri Oct 3 14:24:50 2008
From: bill at cse.ucdavis.edu (Bill Broadley)
Date: Fri, 03 Oct 2008 11:24:50 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E3E69B.4030202@tuffmail.us>
References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
<48E3E69B.4030202@tuffmail.us>
Message-ID: <48E66372.8030707@cse.ucdavis.edu>
Alan Louis Scheinine wrote:
> NiftyOMPI Mitch wrote
>> QDR is interesting... in all likelyhood the
>> QDR game will be optical for any link further away than a single rack.
>> Once IB goes optical there will be a lot of reason to install IB in
>> machine rooms and campus sites that are just out of reach today.
>
> When will IB go optical at a reasonable price?
Folks I know in the IB industry mentioned:
* QDR price life cycle should be similar to the DDR price life cycle.
* QDR Cable lengths over copper are rather poor
* fiber transceivers are finally starting to catch up.
QDR over fiber should be "reasonably priced", here's hoping that the days of
Myrinet 250MB/sec optical cables will return.
Corrections/comments welcome.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From atchley at myri.com Fri Oct 3 14:47:05 2008
From: atchley at myri.com (Scott Atchley)
Date: Fri, 3 Oct 2008 14:47:05 -0400
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E66372.8030707@cse.ucdavis.edu>
References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com>
<48E3E69B.4030202@tuffmail.us> <48E66372.8030707@cse.ucdavis.edu>
Message-ID:
On Oct 3, 2008, at 2:24 PM, Bill Broadley wrote:
> QDR over fiber should be "reasonably priced", here's hoping that the
> days of
> Myrinet 250MB/sec optical cables will return.
>
> Corrections/comments welcome.
I am not in sales and I have no access to pricing besides our list
prices, but I am told that optical QSFP is getting very close to CX-4
when you price NICs and cables (QSFP NICs cost less than CX-4 NICs,
the cables are more, the total package is very close).
Scott
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From coutinho at dcc.ufmg.br Fri Oct 3 15:06:13 2008
From: coutinho at dcc.ufmg.br (Bruno Coutinho)
Date: Fri, 3 Oct 2008 16:06:13 -0300
Subject: [Beowulf] Streamlined standard Linux installation
In-Reply-To:
References:
Message-ID:
2008/10/3 Alan Ward
>
> Hi.
>
> Several days ago there was a thread on using a streamlined standard Linux
> distribution. These days I have been fiddling with the new Debian Lenny
> version (still beta, but it seems not for long).
>
> A "no frills, no X" install with vi, aptitude and a few other rescue tools
> fits comfortably into about 300 Mbytes on an old USB pendrive. Setting a
> maximum of 512 MBytes for a basic cluster node setup seems feasable for some
> applications.
>
> Boot time from GRUB to login prompt is 22s on a 6-year-old Celeron laptop.
>
On ubuntu server you can get less. Thanks to upstart.
The only problem is that login screen gets polluted with starting services
output.
>
> And no - no OpenOffice on this one! ;-)
>
> Cheers,
> -Alan
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Fri Oct 3 15:58:16 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Fri, 3 Oct 2008 12:58:16 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu>
Message-ID: <20081003195816.GB16760@bx9.net>
On Fri, Oct 03, 2008 at 02:17:52AM -0700, Bill Broadley wrote:
> Er, that makes no sense to me. You aren't going to end up with a smaller
> file by encoding a file less efficiently.
I often find that floating-point data doesn't compress much, but that
ASCII representations of the same data compresses to a file smaller
than the binary one. This is with gzip/bzip2, so we're talking
dictionary-based compression -- and it's easy to imagine why this
might be true. I've never seen any float-point-oriented compressor
like the ones specialized for photos (jpeg, etc.)
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Fri Oct 3 16:05:48 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Fri, 3 Oct 2008 13:05:48 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To:
References: <48E39782.4020006@ias.edu>
Message-ID: <20081003200548.GC16760@bx9.net>
On Fri, Oct 03, 2008 at 10:00:05AM -0400, Mark Hahn wrote:
> qdr would be 40 Gb (raw data rate, right? so 4 GB/s before any sort
> of packet overhead, etc.) I don't really see why that's a problem -
> even a memory-constrained current-gen Intel box has about twice that much
> memory bandwidth available. AMD or next-gen-Intel will
> be even less constrained.
Let's say that I'm sending data which is in a cache.
So when the HCA does the DMA operation, all the bytes have to be
flushed from cache to main memory, and then transferred from main
memory to the HCA.
And the system isn't idle while you do this, so these transfers are
less efficient than you might think.
Of course, your "latency and bandwidth" benchmark won't see this
problem, because it only uses a single core, and it sends the same
buffer over and over without touching it.
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From atp at piskorski.com Fri Oct 3 16:05:49 2008
From: atp at piskorski.com (Andrew Piskorski)
Date: Fri, 3 Oct 2008 16:05:49 -0400
Subject: [Beowulf] MonetDB's lightweight compression;
Re: Accelerator for data compressing
In-Reply-To: <48E606B0.3050708@aei.mpg.de>
References: <48E606B0.3050708@aei.mpg.de>
Message-ID: <20081003200549.GB51913@piskorski.com>
On Fri, Oct 03, 2008 at 01:49:04PM +0200, Carsten Aulbert wrote:
> No, quite on the contrary. I would like to use a compressor within a
> pipe to increase the throughput over the network, i.e. to get around the
> ~ 120 MB/s limit.
Carsten, it is probably not directly relevant to you, but you may want
to check out MonetDB, particularly their newer "X100" bleeding edge
R&D version. Among other things, they've published papers with lots
of interesting detail on using super-lightweight software compression
to greatly increase database disk IO bandwith.
Their main software tricks for faster disk IO seemed to be:
One, EXTREMELY lightweight compression schemes - basically table
lookups designed to be as cpu friendly as posible. Two, keep the data
compressed in RAM as well so that you can cache more of the data, and
indeed keep it the compressed until as late in the CPU processing
pipeline as possible. From what I remember, MonetDB/X100 actually
does all decompression solely in the CPU cache, inline with query
processing.
Back c. July 2005, a Sandor Heman, one of the MonetDB guys, looked at
zlib, bzlib2, lzrw, and lzo, to improve database disk IO bandwith, and
claimed that:
"... in general, it is very unlikely that we could achieve any
bandwidth gains with these algorithms. LZRW and LZO might increase
bandwidth on relatively slow disk systems, with bandwidths up to
100MB/s, but this would induce high processing overheads, which
interferes with query execution. On a fast disk system, such as our
350MB/s 12 disk RAID, all the generic algorithms will fail to achieve
any speedup."
http://www.google.com/search?q=MonetDB+LZO+Heman&btnG=Search
http://homepages.cwi.nl/~heman/downloads/msthesis.pdf
"Super-Scalar Database Compression between RAM and CPU Cache"
Back in 2006, the cool new X100 features were not released in MonetDB
proper (which is Open Source), but by now that may have changed.
Lots of links:
http://www.bestechvideos.com/2008/02/21/monetdb-x100-a-very-fast-column-store
ftp://ftp.research.microsoft.com/pub/debull/A05june/issue1.htm
"MonetDB/X100 - A DBMS In The CPU Cache"
http://www.monetdb.nl/
http://homepages.cwi.nl/~mk/MonetDB/
http://sourceforge.net/projects/monetdb/
http://homepages.cwi.nl/~boncz/x100.html
http://www.jsequeira.com/blog/2006/01/17.html
--
Andrew Piskorski
http://www.piskorski.com/
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Fri Oct 3 16:30:21 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Fri, 03 Oct 2008 16:30:21 -0400
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <20081003200548.GC16760@bx9.net>
References: <48E39782.4020006@ias.edu>
<20081003200548.GC16760@bx9.net>
Message-ID: <48E680DD.3@scalableinformatics.com>
Greg Lindahl wrote:
> Of course, your "latency and bandwidth" benchmark won't see this
> problem, because it only uses a single core, and it sends the same
> buffer over and over without touching it.
Yup.
We have a storage test using MPI that has each node generate random
numbers, and passes the buffer around the nodes for IO. The idea is of
course to get a little closer to actual use cases ...
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Shainer at mellanox.com Fri Oct 3 19:55:57 2008
From: Shainer at mellanox.com (Gilad Shainer)
Date: Fri, 3 Oct 2008 16:55:57 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0157FAF6@mtiexch01.mti.com>
Prentice wrote:
> In the ongoing saga that is my new cluster, we were just told
> today that Cisco is no longer manufacturing DDR IB cables,
> which we, uhh, need.
>
> Has DDR IB gone the way of the dodo bird and been supplanted by QDR?
>
> If so, why would anyone spec a brand new cluster with DDR?
>
DDR is still the most used IB solution. I can not speak on Cisco behalf,
but there are many other vendors selling DDR cables - Voltaire,
Mellanox, Gore, QLogic, Molex and many others, including integrators.
Gilad
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Fri Oct 3 20:27:43 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Fri, 03 Oct 2008 20:27:43 -0400
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
<48E2C6C1.60108@neuralbs.com>
<48E3EB2F.2070102@neuralbs.com>
Message-ID: <48E6B87F.6060304@neuralbs.com>
Bogdan Costescu wrote:
> On Wed, 1 Oct 2008, Eric Thibodeau wrote:
>
>> the NFS root approach only does changes on the head node and changed
>> files don't need to be propagated and are accessed on a as-needed
>> basis, this might have significant impacts on large deployments
>
> NFS-root doesn't scale too well, the implementation of NFS in Linux is
> quite chatty.
Someone else responded to this, and expect even more scalability
performance with NFSv4...lots more ;)
>
>> I'll take your word for it that they have a version tracking mechanism.
>
> Take my word that if you're going into larger installations with the
> least amount of non-homogeneity you'll want to at least read about, if
> not use, such mechanisms :-)
>
Well, "large" is relative, if I have a 32k core cluster with all
identical HW, all I really need for this "large" installation is a
single image.. It's the diversity that kills that then requires tight
version/config tracking, even if all the images are managed from a
single point..but that's trivial common sense ;)
Eric
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From coutinho at dcc.ufmg.br Fri Oct 3 21:33:59 2008
From: coutinho at dcc.ufmg.br (Bruno Coutinho)
Date: Fri, 3 Oct 2008 22:33:59 -0300
Subject: [Beowulf] GPU (was Accelerator) for data compressing
In-Reply-To: <6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de>
<5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl>
<48E606B0.3050708@aei.mpg.de>
<48E63E27.6060701@scalableinformatics.com>
<6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl>
Message-ID:
2008/10/3 Vincent Diepeveen
>
> On Oct 3, 2008, at 5:45 PM, Joe Landman wrote:
>
>>
>> GPU as a compression engine? Interesting ...
>>
>> Joe
>>
>>
> For great compression, it's rather hard to get that to work.
> With a lot of RAM some clever guys manage.
>
> GPU has a lot of stream processors, yet little RAM a stream processor.
A tesla have 4GB ram for 240 stream processors. That gives ~16MB for
processor.
But SMID arrays (stream multiptocessors in CUDA) need a lot of threads to
tolerate memory latencies and not get idle.
Something of the order of 512 threads per multiprocessor (tesla have 30), so
we have 4GB divided among 15360, that gives 273k per thread.
>
> Additionally they all have to execute the same code at a bunch of SP's at
> the same time.
>
> So there is a big need for some real clever new algorithm there,
> as the lack of RAM is gonna hurt really bigtime.
You need to get data from machine main memory, compress and send results
back several times.
The bandwidth of pci express today is 8GB/s, so this is the maximum data
rate a gpu can compress.
You can use some tricks like computation and i/o (to main memory)
parallelization, but will be constrained to 8GB/s anyway.
>
>
> Would be a mighty interesting study to get something to work there. It
> allows real
> complicated mathematical functions for PPM functionality.
>
> What's in that GPU soon will be in a CPU anyway, so it benefits the entire
> planet a thing like that.
This will be interesting.
Removing pci express from the path, the gpu will become really a parallel
coprocessor.
With the pressure for a more flexible programming model caused by Larabee,
the coprocessor could become as programable as the main processor and we
will have a processor with few big cores for serial workloads and many cores
for parallel workloads.
>
>
> Where can i ask for funding?
>
> Vincent
>
>
>
> Cheers
>>> Carsten
>>>
>>
>> --
>> Joseph Landman, Ph.D
>> Founder and CEO
>> Scalable Informatics LLC,
>> email: landman at scalableinformatics.com
>> web : http://www.scalableinformatics.com
>> http://jackrabbit.scalableinformatics.com
>> phone: +1 734 786 8423 x121
>> fax : +1 866 888 3112
>> cell : +1 734 612 4615
>>
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From xclski at yahoo.com Sat Oct 4 01:36:56 2008
From: xclski at yahoo.com (Ellis Wilson)
Date: Fri, 3 Oct 2008 22:36:56 -0700 (PDT)
Subject: [Beowulf] GPU (was Accelerator) for data compressing
Message-ID: <499045.95838.qm@web37907.mail.mud.yahoo.com>
Bruno Coutinho wrote:
> You need to get data from machine main memory, compress and send results
> back several times.
> The bandwidth of pci express today is 8GB/s, so this is the maximum data
> rate a gpu can compress.
> You can use some tricks like computation and i/o (to main memory)
> parallelization, but will be constrained to 8GB/s anyway.
Check out PCI Express 2 then, you'll get 16GB/s bandwidth but I'm not
sure all the GPU's are moved that direction yet.
Ellis
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From peter.st.john at gmail.com Sat Oct 4 16:32:57 2008
From: peter.st.john at gmail.com (Peter St. John)
Date: Sat, 4 Oct 2008 16:32:57 -0400
Subject: [Beowulf] OT: the ongoing computer chess championship in China
Message-ID:
The World Computer Chess Championship is onging in China (part of the World
Mind Sports thing). The leaders, about half-way through , are a 40 core
cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5.
There are 10 participants, it's a nine-round round robin. Cluster Toga, also
a cluster, is notable for drawing both of the leaders.
>From http://www.chessbase.com/newsdetail.asp?newsid=4935
Rybka USA Cluster, 40 cores 3.5 4 5.0 4.50 Hiarcs GBR Intel Skulltrail, 8
x 4Ghz 3.5 4 3.0 2.50 Sjeng BEL Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0
2.00 Junior
ISR Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50 The Baron NLD AMD Opteron
270, 4 x 2Ghz 2.0 4 7.0 1.75 Jonny GER Cluster, 16 cores 1.0 4 12.0
2.50 Cluster
Toga GER 24 cores 1.0 3 9.5 3.50 Shredder GER Intel Core 2, 8 x 3.16Ghz 1.0
3 8.0 2.25 Falcon ISR Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00 Mobile
Chess CHN
Nokia 6120c 0.0 4 9.0 0.00
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From peter.st.john at gmail.com Sat Oct 4 17:38:43 2008
From: peter.st.john at gmail.com (Peter St. John)
Date: Sat, 4 Oct 2008 17:38:43 -0400
Subject: [Beowulf] OT: the ongoing computer chess championship in China
In-Reply-To:
References:
Message-ID:
That's the most remarkable crosstable I've ever seen (in Shahrokh's link).
The players are essentially well-ordered; the top beat everyone, 2nd beat
everyone except 1st, 3rd beat everyone except 1 and 2, etc, all the way
down. It makes a upper-trianglular matrix. Remarkable.
I used to give 13 stones to ManyFaces on my own PC. I suspect this beast, on
that cluster, could give me several.
I still give 6 to programs running on just PCs.
(3 stones is about knight odds, in chess terms)
Peter
On 10/4/08, Shahrokh Mortazavi wrote:
>
>
>
> On a related note, at the same tournament, the cluster version of "Many
> Face of Go" has won the Gold medal in both 9X9 and 19X19 categories with no
> defeats. Go is notoriously harder than Chess for computers to play as you
> might now. The results are here:
>
> http://www.grappa.univ-lille3.fr/icga/tournament.php?id=181
>
> incidentally, the code was running on a Windows HPC Server 2008, 32 core
> intel cluster (we helped the author port/tune his MPI code a little).
>
> For some info re challenges in computer Go see:
>
> http://en.wikipedia.org/wiki/Computer_go
>
>
> Shahrokh
>
>
>
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On
> Behalf Of Peter St. John
> Sent: Saturday, October 04, 2008 1:33 PM
> To: beowulf at beowulf.org
> Subject: [Beowulf] OT: the ongoing computer chess championship in China
>
>
> The World Computer Chess Championship is onging in China (part of the World
> Mind Sports thing). The leaders, about half-way through , are a 40 core
> cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5.
> There are 10 participants, it's a nine-round round robin. Cluster Toga, also
> a cluster, is notable for drawing both of the leaders.
>
> >From http://www.chessbase.com/newsdetail.asp?newsid=4935
> Rybka
> USA
> Cluster, 40 cores 3.5 4 5.0 4.50
> Hiarcs
> GBR
> Intel Skulltrail, 8 x 4Ghz 3.5 4 3.0 2.50
> Sjeng
> BEL
> Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0 2.00
> Junior
> ISR
> Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50
> The Baron
> NLD
> AMD Opteron 270, 4 x 2Ghz 2.0 4 7.0 1.75
> Jonny
> GER
> Cluster, 16 cores 1.0 4 12.0 2.50
> Cluster Toga
> GER
> 24 cores 1.0 3 9.5 3.50
> Shredder
> GER
> Intel Core 2, 8 x 3.16Ghz 1.0 3 8.0 2.25
> Falcon
> ISR
> Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00
> Mobile Chess
> CHN
> Nokia 6120c 0.0 4 9.0 0.00
>
>
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From coutinho at dcc.ufmg.br Sat Oct 4 18:26:48 2008
From: coutinho at dcc.ufmg.br (Bruno Coutinho)
Date: Sat, 4 Oct 2008 19:26:48 -0300
Subject: [Beowulf] GPU (was Accelerator) for data compressing
In-Reply-To: <499045.95838.qm@web37907.mail.mud.yahoo.com>
References: <499045.95838.qm@web37907.mail.mud.yahoo.com>
Message-ID:
2008/10/4 Ellis Wilson
> Bruno Coutinho wrote:
> > You need to get data from machine main memory, compress and send results
> > back several times.
> > The bandwidth of pci express today is 8GB/s, so this is the maximum data
> > rate a gpu can compress.
> > You can use some tricks like computation and i/o (to main memory)
> > parallelization, but will be constrained to 8GB/s anyway.
>
> Check out PCI Express 2 then, you'll get 16GB/s bandwidth but I'm not
> sure all the GPU's are moved that direction yet.
>
This is in a x32 slot (that I never saw) in a typical x16 slot pci-e 1.0
make 4GB/s and 2.0 makes 8GB/s.
All nvidia cards from 9 series and ATI from 3xxx series have pci express
2.0.
In chipsets only the newest ones like intel P45 have pci-e 2.0.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hunting at ix.netcom.com Sun Oct 5 02:51:21 2008
From: hunting at ix.netcom.com (Michael Huntingdon)
Date: Sat, 4 Oct 2008 23:51:21 -0700
Subject: [Beowulf] Has DDR IB gone the way of the Dodo?
In-Reply-To: <48E66372.8030707@cse.ucdavis.edu>
Message-ID: <4504CA87400144E09DAD0D6F782D0554@lucky13>
And once thought to be so the "C" word....(ummmm commercial)
Michael Huntingdon
Systems Performance Consultants
Higher Education Account Manager
(408) 294-6811
-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
On Behalf Of Bill Broadley
Sent: Friday, October 03, 2008 11:25 AM
To: alscheinine at tuffmail.us
Cc: NiftyOMPI Mitch; Beowulf Mailing List
Subject: Re: [Beowulf] Has DDR IB gone the way of the Dodo?
Alan Louis Scheinine wrote:
> NiftyOMPI Mitch wrote
>> QDR is interesting... in all likelyhood the
>> QDR game will be optical for any link further away than a single
rack.
>> Once IB goes optical there will be a lot of reason to install IB in
>> machine rooms and campus sites that are just out of reach today.
>
> When will IB go optical at a reasonable price?
Folks I know in the IB industry mentioned:
* QDR price life cycle should be similar to the DDR price life cycle.
* QDR Cable lengths over copper are rather poor
* fiber transceivers are finally starting to catch up.
QDR over fiber should be "reasonably priced", here's hoping that the
days of
Myrinet 250MB/sec optical cables will return.
Corrections/comments welcome.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From eugen at leitl.org Sun Oct 5 05:21:33 2008
From: eugen at leitl.org (Eugen Leitl)
Date: Sun, 5 Oct 2008 11:21:33 +0200
Subject: [Beowulf] Connectionists: NIPS workshop CfP: Parallel
Implementations of Learning Algorithms
Message-ID: <20081005092133.GJ11968@leitl.org>
----- Forwarded message from Dave_Touretzky at cs.cmu.edu -----
From: Dave_Touretzky at cs.cmu.edu
Date: Sun, 05 Oct 2008 00:34:08 -0400
To: connectionists at cs.cmu.edu
Subject: Connectionists: NIPS workshop CfP: Parallel Implementations of
Learning Algorithms
X-Mailer: MH-E 7.4.2; nmh 1.2-20070115cvs; XEmacs 21.5 (beta27)
NIPS08 Workshop Call for Posters:
Parallel Implementations of Learning Algorithms:
What Have You Done For Me Lately?
Overview: Interest in parallel hardware concepts, including multicore,
specialized hardware, and multimachine, has recently increased as
researchers have looked to scale up their concepts to large, complex
models and large datasets. In this workshop, a panel of invited
speakers will present results of investigations into hardware concepts
for accelerating a number of different learning and simulation
algorithms. Additional contributions will be presented in poster
spotlights and a poster session at the end of the one-day workshop.
Our intent is to provide a broad survey of the space of hardware
approaches in order to capture the current state of activity in this
venerable domain of study. Approaches to be covered include silicon,
FPGA, and supercomputer architectures, for applications such as
Bayesian network models of large and complex domains, simulations of
cortex and other brain structures, and large-scale probabilistic
algorithms.
Potential participants include researchers interested in accelerating
their algorithms to handle large datasets, and systems designers
providing such hardware solutions. The oral presentations will
include plenty of time for questions and discussion, and the poster
session at the end of the workshop will afford further opportunities
for interaction among workshop participants.
Confirmed Speakers:
- David Andersen, Carnegie Mellon University
Using a Fast Array of Wimpy Nodes.
- Michael Arnold, Salk Institute
Multi-Scale Modeling in Neuroscience
- Dan Hammerstrom, Portland State University
Nanoelectronics: The Original Positronic Matrix?
- Kenneth Rice, Clemson University
A Neocortex-Inspired Cognitive Model on the Cray XD1
- Robert Thibadeau, Seagate Research
When (And Why) Storage Devices Become Computers
- Additional speakers TBA.
Important Dates:
October 31, 2008 Poster abstract submission deadline
November 7, 2008 Notification of acceptance
December 12 or 13 Workshop at NIPS
How To Submit:
Algorithm developers, hardware developers, and researchers building
large-scale applications are all encouraged to present their work at the
workshop. Send a brief abstract (one page should suffice) in any format
describing the approach and results you wish to present in poster form
at the meeting to. David S. Touretzky, Carnegie Mellon University,
dst at cs.cmu.edu. Submissions are due by October 31; decisions will be
announced by November 7. Authors of accepted posters will be asked to
give a 2 minute spotlight presentation before the start of the poster
session.
Workshop Organizing Committee:
Robert Thibadeau, Seagate Research
Dan Hammerstrom, Portland State University
David Touretzky, Carnegie Mellon University
Tom Mitchell, Carnegie Mellon University
----- End forwarded message -----
--
Eugen* Leitl leitl http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From eugen at leitl.org Sun Oct 5 11:30:34 2008
From: eugen at leitl.org (Eugen Leitl)
Date: Sun, 5 Oct 2008 17:30:34 +0200
Subject: [Beowulf] FY;) the Helmer cluster
Message-ID: <20081005153034.GU11968@leitl.org>
A Linux cluster built in a Helmer IKEA cabinet:
http://helmer.sfe.se/
Here's another version of it:
http://obscuredclarity.blogspot.com/2008/09/24-core-linux-cluster-in-2999-case-from.html
Cooling looks a bit suboptimal.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Sun Oct 5 15:33:46 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Sun, 5 Oct 2008 21:33:46 +0200
Subject: [Beowulf] IKEA priced supercomputer - the Helmer cluster
In-Reply-To: <20081005153034.GU11968@leitl.org>
References: <20081005153034.GU11968@leitl.org>
Message-ID: <867A068C-7E3B-4D1B-992E-A6F6840B70A2@xs4all.nl>
This is great stuff to read :)
Those Swedes can do everything with wood it seems :)
Especially the IKEA idea and the height comparision with the Michael
Johnson shoe gave me some great laughs :)
Maybe you can modify the drawers of the ikea box and then have also
real easy maintenance with them.
Without kidding, it's a great idea, and even better that it got
documented. I should do that too in case i win the lottery
and can make a cluster here.
Do i see it correct that maintenance an entire mainboard can get
shifted to outside?
How can maintenance be done?
What i kind of miss is how to get the iron backplates where the fans
and psu's fit in and how a possible highend network,
in my case 2nd hand can be mounted into it.
More pictures from how it looks like would be fantastic for those who
want to take a look at how to themselves make
a bunch of nodes in this manner; simply how it looks like.
It seems to me it can store a lot of ATX mainboards in real cheap
manner. The way
the cooling gets blown into the case doesn't seems very professional
to me. Does it limit maintenance to the motherboards?
Vincent
On Oct 5, 2008, at 5:30 PM, Eugen Leitl wrote:
>
> A Linux cluster built in a Helmer IKEA cabinet:
>
> http://helmer.sfe.se/
>
> Here's another version of it:
>
> http://obscuredclarity.blogspot.com/2008/09/24-core-linux-cluster-
> in-2999-case-from.html
>
> Cooling looks a bit suboptimal.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From niftyompi at niftyegg.com Sun Oct 5 20:15:14 2008
From: niftyompi at niftyegg.com (Nifty niftyompi Mitch)
Date: Sun, 5 Oct 2008 17:15:14 -0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <20081003195816.GB16760@bx9.net>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <20081003195816.GB16760@bx9.net>
Message-ID: <20081006001514.GA4502@compegg.wr.niftyegg.com>
On Fri, Oct 03, 2008 at 12:58:16PM -0700, Greg Lindahl wrote:
> On Fri, Oct 03, 2008 at 02:17:52AM -0700, Bill Broadley wrote:
>
> > Er, that makes no sense to me. You aren't going to end up with a smaller
> > file by encoding a file less efficiently.
>
> I often find that floating-point data doesn't compress much, but that
> ASCII representations of the same data compresses to a file smaller
> than the binary one. This is with gzip/bzip2, so we're talking
> dictionary-based compression -- and it's easy to imagine why this
> might be true. I've never seen any float-point-oriented compressor
> like the ones specialized for photos (jpeg, etc.)
Floating point numbers not compressing makes sense sense to me.
The noise of the LSB and layout would mask redunancy in the exponent and other
fields. Since most compression tools are byte (character and string)
oriented in their design the ASCII version could expose the right stuff.
It does point out a gap in the compression tool suite that I think the
hard data folks might look at..... interesting stuff.
--
T o m M i t c h e l l
Found me a new hat, now what?
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From gerry.creager at tamu.edu Mon Oct 6 09:11:02 2008
From: gerry.creager at tamu.edu (Gerry Creager)
Date: Mon, 06 Oct 2008 08:11:02 -0500
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
Message-ID: <48EA0E66.6060403@tamu.edu>
John Hearns wrote:
>
>
> 2008/10/1 Donald Becker >
>
>
>
> It's foreseeable that holding an 8GB install
> image in memory will be trivial, but that will be a few years in the
> future, not today. And we will need better VM and PTE management to
> make
> it efficient.
>
>
>
> Hmmmm.... can I forsee Puppy Linux HPC Edition ????
> http://www.puppylinux.org/
>
>
> Being half serious here, is it worth trying to get one of these
> slimmed-down distros to the state where it will run an HPC job?
> Oh, and in addition to a barebones install for our contemplative
> cluster, they have a onebones install which cuts out the GUI (cue more
> dog puns).
>
> http://www.puppylinux.com/pfs/ looks interesting...
Starts to sound like Rocks. Which I'll finish installing on a small
cluster tomorrow and will form more opinions then. First impressions
(really second: I've had another small cluster up with it for 3 years)
is that it's seamless for the scientists once they figure out where
their apps are. I've been pretty happy with it: It's been mostly "set
and forget".
Bogdan, we're in a similar position, save I also do some of the science.
Our latest cluster serves users in Atmospheric Science, Chemistry,
Math, Statistics, Agriculture & Life Sciences, Computer Science,
Oceanography and Nuclear (or was that "nuculeer"?) engineering... Most
are somewhat astute to computation but not one rises to the point of
"computational scientist" in being able to combine extensive computer
science/engineering knowledge with their much more important discipline
knowledge. Hiding the details from them, but meeting their needs at the
same time is a constant challenge.
--
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From smortaz at exchange.microsoft.com Sat Oct 4 17:08:36 2008
From: smortaz at exchange.microsoft.com (Shahrokh Mortazavi)
Date: Sat, 4 Oct 2008 14:08:36 -0700
Subject: [Beowulf] OT: the ongoing computer chess championship in China
In-Reply-To:
References:
Message-ID:
On a related note, at the same tournament, the cluster version of "Many Face of Go" has won the Gold medal in both 9X9 and 19X19 categories with no defeats. Go is notoriously harder than Chess for computers to play as you might now. The results are here:
http://www.grappa.univ-lille3.fr/icga/tournament.php?id=181
incidentally, the code was running on a Windows HPC Server 2008, 32 core intel cluster (we helped the author port/tune his MPI code a little).
For some info re challenges in computer Go see:
http://en.wikipedia.org/wiki/Computer_go
Shahrokh
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Peter St. John
Sent: Saturday, October 04, 2008 1:33 PM
To: beowulf at beowulf.org
Subject: [Beowulf] OT: the ongoing computer chess championship in China
The World Computer Chess Championship is onging in China (part of the World Mind Sports thing). The leaders, about half-way through , are a 40 core cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5. There are 10 participants, it's a nine-round round robin. Cluster Toga, also a cluster, is notable for drawing both of the leaders.
>From http://www.chessbase.com/newsdetail.asp?newsid=4935
Rybka
USA
Cluster, 40 cores 3.5 4 5.0 4.50
Hiarcs
GBR
Intel Skulltrail, 8 x 4Ghz 3.5 4 3.0 2.50
Sjeng
BEL
Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0 2.00
Junior
ISR
Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50
The Baron
NLD
AMD Opteron 270, 4 x 2Ghz 2.0 4 7.0 1.75
Jonny
GER
Cluster, 16 cores 1.0 4 12.0 2.50
Cluster Toga
GER
24 cores 1.0 3 9.5 3.50
Shredder
GER
Intel Core 2, 8 x 3.16Ghz 1.0 3 8.0 2.25
Falcon
ISR
Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00
Mobile Chess
CHN
Nokia 6120c 0.0 4 9.0 0.00
Peter
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From i.n.kozin at googlemail.com Sun Oct 5 16:41:20 2008
From: i.n.kozin at googlemail.com (Igor Kozin)
Date: Sun, 5 Oct 2008 21:41:20 +0100
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To: <20081005153034.GU11968@leitl.org>
References: <20081005153034.GU11968@leitl.org>
Message-ID:
lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
http://helmer3.sfe.se/
2008/10/5 Eugen Leitl
>
> A Linux cluster built in a Helmer IKEA cabinet:
>
> http://helmer.sfe.se/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From jason at acm.org Mon Oct 6 12:50:30 2008
From: jason at acm.org (Jason Riedy)
Date: Mon, 06 Oct 2008 12:50:30 -0400
Subject: [Beowulf] Re: Accelerator for data compressing
In-Reply-To: <20081003195816.GB16760@bx9.net> (Greg Lindahl's message of "Fri,
3 Oct 2008 12:58:16 -0700")
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48E5712E.7070504@cse.ucdavis.edu>
<692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl>
<48E5E340.6080408@cse.ucdavis.edu> <20081003195816.GB16760@bx9.net>
Message-ID: <87y711wtux.fsf@sparse.dyndns.org>
And Greg Lindahl writes:
> I often find that floating-point data doesn't compress much, but that
> ASCII representations of the same data compresses to a file smaller
> than the binary one.
However, some care is necessary if you're using decimal output rather
than hex. I've run into problems with the identical decimal (in ASCII)
number converting to different binary numbers. The difference is only
in the last bit, but that ends up being important for my uses.
Sometimes the cause is an actual bug in the conversion, but most times
the decimal version provided *more* than the number of decimal digits
expected for the binary format. For example, some people store 20
digits rather than 17 for decimal. Different systems handle the extra
digits differently. sigh. And remember that 754-1985 did *not* require
correct conversion for all numbers, so some people have not provided
correct conversion. (The new 754-2008 *does* require correct
conversion.)
But there's C99-style hex output. Makes my life much easier. And
conversions are relatively fast, unlike the conversions to/from decimal.
Jason
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Mon Oct 6 15:55:42 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Mon, 06 Oct 2008 15:55:42 -0400
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To:
References: <20081005153034.GU11968@leitl.org>
Message-ID: <48EA6D3E.8070804@neuralbs.com>
Igor Kozin wrote:
> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
> http://helmer3.sfe.se/
LOL, I say we put it up at my cottage, electricity is cheap in Quebec
and we have a massive lake besides it :)
/me bypasses exit heat to warm up house.
Now, doesn't anyone seem to recognize a Crayish look to this cluster... :P
>
>
> 2008/10/5 Eugen Leitl >
>
>
> A Linux cluster built in a Helmer IKEA cabinet:
>
> http://helmer.sfe.se/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Mon Oct 6 18:57:33 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Tue, 7 Oct 2008 00:57:33 +0200
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To:
References: <20081005153034.GU11968@leitl.org>
Message-ID:
Nah if you just want to run 1 application at 4 pflop that should be
no problem
to design some special hardware for it for $0.9M
Vincent
On Oct 5, 2008, at 10:41 PM, Igor Kozin wrote:
> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
> http://helmer3.sfe.se/
>
>
> 2008/10/5 Eugen Leitl
>
> A Linux cluster built in a Helmer IKEA cabinet:
>
> http://helmer.sfe.se/
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Dan.Kidger at quadrics.com Tue Oct 7 03:34:22 2008
From: Dan.Kidger at quadrics.com (Dan.Kidger at quadrics.com)
Date: Tue, 7 Oct 2008 08:34:22 +0100
Subject: [Beowulf] large MPI adopters
In-Reply-To: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
Andrea,
MPI is of course used by many applications running on commercial clusters.
Two obvious examples are computational chemistry by the drug companies
and computational fluid dynamics (CFD) for aerospace companies and F1 design teams.
These are all long-term 'traditional' uses of MPI for scientific/engineering codes.
Is this what you are asking? Or are you thinking of non-traditional uses in say computational finance or gaming sites?
Daniel
-------------------------------------------------------------
Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
One Bridewell St., Mobile: +44 (0)779 209 1851
Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
----------------------- www.quadrics.com --------------------
-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Andrea Di Blas
Sent: 13 August 2008 00:37
To: beowulf at beowulf.org
Subject: [Beowulf] large MPI adopters
hello,
I am curious about what companies, besides the national labs of course,
use any implementation of MPI to support large applications of any kind,
whether only internally (like mapreduce for google, for example) or not.
does anybody know of any cases?
thank you and best regards,
andrea
--
Andrea Di Blas, UCSC
School of Engineering
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Tue Oct 7 07:37:22 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Tue, 7 Oct 2008 13:37:22 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <48EA0E66.6060403@tamu.edu>
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
<48EA0E66.6060403@tamu.edu>
Message-ID:
On Mon, 6 Oct 2008, Gerry Creager wrote:
> Bogdan, we're in a similar position, save I also do some of the
> science.
I (try to) do some of the science too, so the similarity is even
better :-)
> Hiding the details from them, but meeting their needs at the same
> time is a constant challenge.
Thanks for sharing all these details, but I would be more interested
to know about the technical solutions that you have applied. How do
you deal with the various software packages that require specific
conditions to run ? Do you have separate clusters for specific
purposes/software and only direct users to them ?
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Tue Oct 7 07:49:19 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Tue, 7 Oct 2008 12:49:19 +0100
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To:
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
<48EA0E66.6060403@tamu.edu>
Message-ID: <9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com>
2008/10/7 Bogdan Costescu
>
> Thanks for sharing all these details, but I would be more interested to
> know about the technical solutions that you have applied. How do you deal
> with the various software packages that require specific conditions to run ?
> Do you have separate clusters for specific purposes/software and only direct
> users to them ?
>
>
>
If we're talking about different interconnects to run specific jobs, then
its quite easy to build a cluster with (say) a few racks of Myrinet equipped
nodes. You just set up your job scheduler such that scientists needing to
use Myrinet can request them.
Or by "conditions" do you mean diferent libraries or compilers?
John Hearns
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Tue Oct 7 08:58:43 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Tue, 7 Oct 2008 14:58:43 +0200
Subject: [Beowulf] large MPI adopters
In-Reply-To: <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
<0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
Message-ID: <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
Hi Dan,
I'd rather guess his question is more: "i want a big list of 10000
companies that are actively using clusters".
That list is not there and if it was there it is utmost secretly
compiled by some NSA type instance.
I'm now speaking more for western Europe, not necessarily for USA:
The research that companies perform is usually applied research: the
minimum requirement to produce a new product,
or when forced at gunpoint by government for safety reasons.
For fundamental research usually most computational resources get
thrown into battle, which is for nearly 100% sponsored
by government be it directly or indirectly.
In fact there is entire industries, especially clean energy research,
that is 100% paid by government subsidies.
Investigations after new products, in 99% of cases get funded by
government; usually the name of a big company is behind it,
but it's still government dudes who sit in that company doing the
research. What matters of course is who pays it.
The exception is persons who have their own company and are some sort
of big expert at a certain area. Take me.
I want to do some research after learning algorithms, at a very high
level (so far they didn't even conclude the right
value for material for chess, let alone that they were useful at
anything other than debugging your code, which is a big
important thing by the way).
Without getting the help of someone working at an university to run
such a thing overnight at a 160-200 cores,
i would of course only be able to run it at a single pc. Now i don't
get any money for all this from government,
so the big question is whether i'm supported by government then or not.
What i do know is that all public research here at government level
is major shit. This is explainable by the fact that a field is nonstop
on the move, only those who have been busy in a field for 10+ years
have any clue what happens there and are the first to try something;
the PHD's are simply busy with inferior software meanwhile having
massive hardware that's idle. A big problem is the nonpublishing of
accomplishments; First of all i'm not publishing *how* i'm doing the
learning. When some university offers me a job, i might publish things,
i'm not getting paid for publications. Why help "the competition"?
What might get published is the guy who helps me; one tiny thing of
all this he'll publish probably. That's maybe 1/100 of the conclusions
drawn.
Only the NSA type organisations know as they all spy on the internet,
and you BET that all big
countries know exactly what i'm doing, they all tap the internet like
crazy there.
If 1 researcher is real brilliant and CAN need some big computation,
there is at least a 100 spies who do know something of that field,
and have to interpret the tapped data. What i do not know, as i don't
work for any of the N*SA type organisations, is in how far *those*
verify things using massive hardware.
Most researchers have no clue how big the spying and anti-spying is;
most might get total scared if they'd realized how much they
get protected.
It is easier to steal it than to find an Einstein in your own nation
figuring it out.
That is the principle that every nation uses; CIA is a very tiny
organisation compared to what other nations have,
note that the weird thing is that CIA is notorious for violating
agreements with other nations (not spying in friendly nations,
to give an example; and also passing on information to their
companies that they got from the information stream they
got from friendly nations spying onto their own companies).
The fact that i write this down already means i have not been
employed in that field nor am; otherwise i would not even *mention*
the word.
Or to quote someone who works in that field when i said that there
gotta be an awful lot of civil servants in Netherlands busy in that
field,
as there is 3+ million tourists a year in Netherlands:
"we have tourism in Spain also".
Yet nothing on the internet is safe, that's the basic problem here.
It's too easy to hack internet lines. 1024 bits or 2048 RSA or
something that
gets used for SSH?
1024 bits is too easy to crack for 1 organisation of each country by
simply throwing specialized hardware at it.
2048 bits RSA is a tad harder, but also not a problem.
128 bits AES?
No manner to attack it is public known (would only deliver $20k,
which is too little anyway, for the risk you take publishing a method).
Yet even if there would be some sort of randomized GA-algorithm that
needs some sort of Pollard-Rho order O( 2 ^ 0.25n ),
then a simple dedicated cpu can already crack it handsdown.
Obviously most companies are not busy spamming the net what they do
with clusters or do not do.
Keeping it secret for their competitors is just too important, yet if
you ask me i feel big companies do real little
research in areas that do not lead directly to products of them.
There might be 1 or 2 exceptions, like oil companies,
but the question is in how far researchers there can be seen as
employees of that company as they get indirectly
sponsored by government whose interest in everything that happens
with oil is real big.
If you ask me however, way too little fundamental research gets
sponsored by government.
If you see just in a tiny nation like netherlands; the number of
'researchers' that work for government is just a very small part of
the number of professors. It's like 1 researcher for each 2
professors; PHD's not counted.
Note this is also what a few studies recently showed.
Just the comparision to intelligence agencies which can put into
action each one of them quarter of a million to millions of people,
and who hire real easy people be it direct or indirect, it is obvious
that the saying: "Better well stolen than bad invented",
gets written in just too big capitals.
As we're speaking of a 1000 individuals in a nation having 16.5
million inhabitants, and only very few of those have
time to devote a big part of their time to do research; maybe 5% is?
Most are too busy with meetings and other organisational work and
posting on the internet.
Realize that same nation also has over a 1000 companies with
more than 1000 employees and half a million charity organisations.
That really lets fundamental research look like something that gets
total neglected.
Vincent
On Oct 7, 2008, at 9:34 AM, wrote:
> Andrea,
>
> MPI is of course used by many applications running on commercial
> clusters.
> Two obvious examples are computational chemistry by the drug companies
> and computational fluid dynamics (CFD) for aerospace companies and
> F1 design teams.
>
> These are all long-term 'traditional' uses of MPI for scientific/
> engineering codes.
>
> Is this what you are asking? Or are you thinking of non-traditional
> uses in say computational finance or gaming sites?
>
> Daniel
>
> -------------------------------------------------------------
> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
> One Bridewell St., Mobile: +44 (0)779 209 1851
> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
> ----------------------- www.quadrics.com --------------------
>
>
>
>
> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-
> bounces at beowulf.org] On Behalf Of Andrea Di Blas
> Sent: 13 August 2008 00:37
> To: beowulf at beowulf.org
> Subject: [Beowulf] large MPI adopters
>
> hello,
>
>
> I am curious about what companies, besides the national labs of
> course,
> use any implementation of MPI to support large applications of any
> kind,
> whether only internally (like mapreduce for google, for example) or
> not.
>
> does anybody know of any cases?
> thank you and best regards,
>
>
> andrea
>
>
>
>
> --
> Andrea Di Blas, UCSC
> School of Engineering
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Tue Oct 7 09:11:37 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Tue, 7 Oct 2008 15:11:37 +0200 (CEST)
Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk
In-Reply-To: <9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com>
References:
<9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com>
<48EA0E66.6060403@tamu.edu>
<9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com>
Message-ID:
On Tue, 7 Oct 2008, John Hearns wrote:
> If we're talking about different interconnects to run specific jobs
Most of my previous messages in this thread were related to having to
run multiple distributions to cater for the needs of specific
application software which would not run or would not be supported if
run on a distribution other than the one specified by the ISV.
The question of interconnect which limits running of specific software
is a similar, but different question. I also had to deal with software
that uses f.e. GlobalArrays/ARMCI for which only GM but not MX is
supported on Myrinet cards.
> You just set up your job scheduler such that scientists needing to
> use Myrinet can request them.
Yes, this is already in use here for some time.
> Or by "conditions" do you mean diferent libraries or compilers?
Different libraries is partly covered already when talking about
different distributions. Different compilers... hmm, I don't want to
start on this subject, I've already written enough in this thread :-)
But can f.e. someone shed some light as to why PathScale compilers are
not supported on Debian or Ubuntu (LTS) ?
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From gerry.creager at tamu.edu Tue Oct 7 09:52:54 2008
From: gerry.creager at tamu.edu (Gerry Creager)
Date: Tue, 07 Oct 2008 08:52:54 -0500
Subject: [Beowulf] large MPI adopters
In-Reply-To: <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
Message-ID: <48EB69B6.5080602@tamu.edu>
I'm top posting, 'cause everyone else is. It's easier.
Vincent, there are a lot of folks interested in computer and
infrastructure security. They are interested in making it more secure
and interested in exploiting its holes. And these are the "good guys".
Stick to chess. Don't blather about security, an area you appear to
be less aware of than the board games. I don't tell you how to be a
chess grand master, and you don't tell me about security from a purely
speculative view... OK?
Andrea, a lot of scientific computation uses big MPI. Off the top of my
head, and based on examples I'm personally aware of (but don't work in
directly), seismic and reservoir flow models are big MPI operations.
Think "oil company", e.g., Shell, Chevron, Exxon-Mobil, etc., I do know
that, in Houston, all employ big clusters running MPI on a regular
basis. Quantum chemistry is often run as an MPI job. Weather modeling
is a big MPI user, as are ocean and coastal models run at universities,
government facilities (not solely national labs) and corporate entities.
NASA does a fair bit of MPI work... where did Beowulfery start (Don,
that's a cue)? We have a group using MPI applications in genomics
research, and others using it for physical chemistry modeling, LHC data
operations/reduction/interpretation, and nuclear systems prognostication.
The list isn't small: As Dan mentions, the drug companies are big users,
although there's a fair bit of openMPI operation in their codes, too.
CFD in aerospace, and auto manufacturers. The US military almost
certainly does parallel processing involving both message passing and
Monte Carlo simulations. I've written code that solves physical and
satellite geodesy problems using PVM (but I was young and that was a
long time ago; today I'd use MPI); a variant of that was apparently
moved to the production world after I changed positions and couldn't
support it anymore.
I'm not sure to tell you who the biggest corporate users are. It likely
depends on one's exposure to even have an opinion. However, for the
biggest corporate users I'm aware of in my immediate area, Schlumberger,
Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
obviously a very incomplete list.
Gerry
Vincent Diepeveen wrote:
> Hi Dan,
>
> I'd rather guess his question is more: "i want a big list of 10000
> companies that are actively using clusters".
> That list is not there and if it was there it is utmost secretly
> compiled by some NSA type instance.
>
> I'm now speaking more for western Europe, not necessarily for USA:
>
> The research that companies perform is usually applied research: the
> minimum requirement to produce a new product,
> or when forced at gunpoint by government for safety reasons.
>
> For fundamental research usually most computational resources get thrown
> into battle, which is for nearly 100% sponsored
> by government be it directly or indirectly.
>
> In fact there is entire industries, especially clean energy research,
> that is 100% paid by government subsidies.
> Investigations after new products, in 99% of cases get funded by
> government; usually the name of a big company is behind it,
> but it's still government dudes who sit in that company doing the
> research. What matters of course is who pays it.
>
> The exception is persons who have their own company and are some sort of
> big expert at a certain area. Take me.
> I want to do some research after learning algorithms, at a very high
> level (so far they didn't even conclude the right
> value for material for chess, let alone that they were useful at
> anything other than debugging your code, which is a big
> important thing by the way).
>
> Without getting the help of someone working at an university to run such
> a thing overnight at a 160-200 cores,
> i would of course only be able to run it at a single pc. Now i don't get
> any money for all this from government,
> so the big question is whether i'm supported by government then or not.
>
> What i do know is that all public research here at government level is
> major shit. This is explainable by the fact that a field is nonstop
> on the move, only those who have been busy in a field for 10+ years have
> any clue what happens there and are the first to try something;
> the PHD's are simply busy with inferior software meanwhile having
> massive hardware that's idle. A big problem is the nonpublishing of
> accomplishments; First of all i'm not publishing *how* i'm doing the
> learning. When some university offers me a job, i might publish things,
> i'm not getting paid for publications. Why help "the competition"?
> What might get published is the guy who helps me; one tiny thing of all
> this he'll publish probably. That's maybe 1/100 of the conclusions
> drawn.
> Only the NSA type organisations know as they all spy on the internet,
> and you BET that all big
> countries know exactly what i'm doing, they all tap the internet like
> crazy there.
>
> If 1 researcher is real brilliant and CAN need some big computation,
> there is at least a 100 spies who do know something of that field,
> and have to interpret the tapped data. What i do not know, as i don't
> work for any of the N*SA type organisations, is in how far *those*
> verify things using massive hardware.
>
> Most researchers have no clue how big the spying and anti-spying is;
> most might get total scared if they'd realized how much they
> get protected.
>
> It is easier to steal it than to find an Einstein in your own nation
> figuring it out.
> That is the principle that every nation uses; CIA is a very tiny
> organisation compared to what other nations have,
> note that the weird thing is that CIA is notorious for violating
> agreements with other nations (not spying in friendly nations,
> to give an example; and also passing on information to their companies
> that they got from the information stream they
> got from friendly nations spying onto their own companies).
>
> The fact that i write this down already means i have not been employed
> in that field nor am; otherwise i would not even *mention* the word.
>
> Or to quote someone who works in that field when i said that there gotta
> be an awful lot of civil servants in Netherlands busy in that field,
> as there is 3+ million tourists a year in Netherlands:
> "we have tourism in Spain also".
>
> Yet nothing on the internet is safe, that's the basic problem here. It's
> too easy to hack internet lines. 1024 bits or 2048 RSA or something that
> gets used for SSH?
>
> 1024 bits is too easy to crack for 1 organisation of each country by
> simply throwing specialized hardware at it.
> 2048 bits RSA is a tad harder, but also not a problem.
>
> 128 bits AES?
>
> No manner to attack it is public known (would only deliver $20k, which
> is too little anyway, for the risk you take publishing a method).
> Yet even if there would be some sort of randomized GA-algorithm that
> needs some sort of Pollard-Rho order O( 2 ^ 0.25n ),
> then a simple dedicated cpu can already crack it handsdown.
>
> Obviously most companies are not busy spamming the net what they do with
> clusters or do not do.
> Keeping it secret for their competitors is just too important, yet if
> you ask me i feel big companies do real little
> research in areas that do not lead directly to products of them. There
> might be 1 or 2 exceptions, like oil companies,
> but the question is in how far researchers there can be seen as
> employees of that company as they get indirectly
> sponsored by government whose interest in everything that happens with
> oil is real big.
>
> If you ask me however, way too little fundamental research gets
> sponsored by government.
> If you see just in a tiny nation like netherlands; the number of
> 'researchers' that work for government is just a very small part of
> the number of professors. It's like 1 researcher for each 2 professors;
> PHD's not counted.
>
> Note this is also what a few studies recently showed.
>
> Just the comparision to intelligence agencies which can put into action
> each one of them quarter of a million to millions of people,
> and who hire real easy people be it direct or indirect, it is obvious
> that the saying: "Better well stolen than bad invented",
> gets written in just too big capitals.
>
> As we're speaking of a 1000 individuals in a nation having 16.5 million
> inhabitants, and only very few of those have
> time to devote a big part of their time to do research; maybe 5% is?
>
> Most are too busy with meetings and other organisational work and
> posting on the internet.
> Realize that same nation also has over a 1000 companies with
> more than 1000 employees and half a million charity organisations.
>
> That really lets fundamental research look like something that gets
> total neglected.
>
> Vincent
>
> On Oct 7, 2008, at 9:34 AM,
> wrote:
>
>> Andrea,
>>
>> MPI is of course used by many applications running on commercial
>> clusters.
>> Two obvious examples are computational chemistry by the drug companies
>> and computational fluid dynamics (CFD) for aerospace companies and F1
>> design teams.
>>
>> These are all long-term 'traditional' uses of MPI for
>> scientific/engineering codes.
>>
>> Is this what you are asking? Or are you thinking of non-traditional
>> uses in say computational finance or gaming sites?
>>
>> Daniel
>>
>> -------------------------------------------------------------
>> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
>> One Bridewell St., Mobile: +44 (0)779 209 1851
>> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
>> ----------------------- www.quadrics.com --------------------
>>
>>
>>
>>
>> -----Original Message-----
>> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
>> On Behalf Of Andrea Di Blas
>> Sent: 13 August 2008 00:37
>> To: beowulf at beowulf.org
>> Subject: [Beowulf] large MPI adopters
>>
>> hello,
>>
>>
>> I am curious about what companies, besides the national labs of course,
>> use any implementation of MPI to support large applications of any kind,
>> whether only internally (like mapreduce for google, for example) or not.
>>
>> does anybody know of any cases?
>> thank you and best regards,
>>
>>
>> andrea
>>
>>
>>
>>
>> --
>> Andrea Di Blas, UCSC
>> School of Engineering
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
--
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From tom.elken at qlogic.com Tue Oct 7 11:01:58 2008
From: tom.elken at qlogic.com (Tom Elken)
Date: Tue, 7 Oct 2008 08:01:58 -0700
Subject: [Beowulf] large MPI adopters
In-Reply-To: <48EB69B6.5080602@tamu.edu>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com><3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org>
> [mailto:beowulf-bounces at beowulf.org] On Behalf Of Gerry Creager
> The list isn't small: As Dan mentions, the drug companies are
> big users,
> although there's a fair bit of openMPI operation in their codes, too.
Gerry,
I'm sure you meant
OpenMP
(standard shared-memory threaded parallelism)
So easy for fingers to type "MPI".
Even though MPI has far more usage than OpenMP, I have to stick up for
it as a former shared-memory advocate from my SGI days and as a member
of the OpenMP ARB.
-Tom
> CFD in aerospace, and auto manufacturers. The US military almost
> certainly does parallel processing involving both message passing and
> Monte Carlo simulations. I've written code that solves physical and
> satellite geodesy problems using PVM (but I was young and that was a
> long time ago; today I'd use MPI); a variant of that was apparently
> moved to the production world after I changed positions and couldn't
> support it anymore.
>
> I'm not sure to tell you who the biggest corporate users are.
> It likely
> depends on one's exposure to even have an opinion. However, for the
> biggest corporate users I'm aware of in my immediate area,
> Schlumberger,
> Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
> obviously a very incomplete list.
>
>
> Gerry
>
>> From: beowulf-bounces at beowulf.org
> [mailto:beowulf-bounces at beowulf.org]
> >> On Behalf Of Andrea Di Blas
> >> Sent: 13 August 2008 00:37
> >> To: beowulf at beowulf.org
> >> Subject: [Beowulf] large MPI adopters
> >>
> >> hello,
> >>
> >>
> >> I am curious about what companies, besides the national
> labs of course,
> >> use any implementation of MPI to support large
> applications of any kind,
> >> whether only internally (like mapreduce for google, for
> example) or not.
> >>
> >> does anybody know of any cases?
> >> thank you and best regards,
> >>
> >>
> >> andrea
> >>
> >>
> >>
> >>
> >> --
> >> Andrea Di Blas, UCSC
> >> School of Engineering
> >> _______________________________________________
> >> Beowulf mailing list, Beowulf at beowulf.org
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> >>
> >> _______________________________________________
> >> Beowulf mailing list, Beowulf at beowulf.org
> >> To change your subscription (digest mode or unsubscribe) visit
> >> http://www.beowulf.org/mailman/listinfo/beowulf
> >>
> >
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> Gerry Creager -- gerry.creager at tamu.edu
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe)
> visit http://www.beowulf.org/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Tue Oct 7 11:33:52 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Tue, 7 Oct 2008 17:33:52 +0200
Subject: [Beowulf] large MPI adopters
In-Reply-To: <48EB69B6.5080602@tamu.edu>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
Message-ID:
On Oct 7, 2008, at 3:52 PM, Gerry Creager wrote:
> I'm top posting, 'cause everyone else is. It's easier.
>
>
> Andrea, a lot of scientific computation uses big MPI. Off the top
> of my head, and based on examples I'm personally aware of (but
> don't work in directly), seismic and reservoir flow models are big
> MPI operations. Think "oil company", e.g., Shell, Chevron, Exxon-
> Mobil, etc., I do know that, in Houston, all employ big clusters
> running MPI on a regular basis. Quantum chemistry is often run as
> an MPI job. Weather modeling is a big MPI user, as are ocean and
> coastal models run at universities, government facilities (not
> solely national labs) and corporate entities. NASA does a fair bit
> of MPI work... where did Beowulfery start (Don, that's a cue)? We
> have a group using MPI applications in genomics research, and
> others using it for physical chemistry modeling, LHC data
> operations/reduction/interpretation, and nuclear systems
> prognostication.
>
> The list isn't small: As Dan mentions, the drug companies are big
> users, although there's a fair bit of openMPI operation in their
> codes, too. CFD in aerospace, and auto manufacturers. The US
> military almost certainly does parallel processing involving both
> message passing and Monte Carlo simulations. I've written code
> that solves physical and satellite geodesy problems using PVM (but
> I was young and that was a long time ago; today I'd use MPI); a
> variant of that was apparently moved to the production world after
> I changed positions and couldn't support it anymore.
>
> I'm not sure to tell you who the biggest corporate users are. It
> likely depends on one's exposure to even have an opinion. However,
> for the biggest
You raise an interesting question.
Military type things not counted, biggest corporate users of generic
cpu's is car industry.
This is quite logical: safety is everything with cars and a car has
far over 50+ processors.
Not seldom 100 nowadays.
Drug type companies eat typical 0.5% system time.
They are far behind, which is not so logical; i would consider it old
fashioned to do experiments
on monkeys (Bar-ilan brain experiments on monkeys rings a bell?), so
doing everything in software is far more logical.
The big question is: should one just look to how many generic cpu's
one uses?
When someone is really in big need for big calculation power, one
makes dedicated cpu's for it.
It may come...
In most embedded devices that badly need high levels of encryption,
typically a coprocessor gets put in to do the job.
Such co processor is really mighty if one would compare it with how
many Tflop such a thing is when putting in generic cpu's.
If you look at it from that viewpoint, ISPs are worlds biggest
corporate users of hardware.
Now go back in your corner and secure the internet :)
> corporate users I'm aware of in my immediate area, Schlumberger,
> Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
> obviously a very incomplete list.
>
> Gerry
>
> Vincent Diepeveen wrote:
>> Hi Dan,
>> I'd rather guess his question is more: "i want a big list of 10000
>> companies that are actively using clusters".
>> That list is not there and if it was there it is utmost secretly
>> compiled by some NSA type instance.
>> I'm now speaking more for western Europe, not necessarily for USA:
>> The research that companies perform is usually applied research:
>> the minimum requirement to produce a new product,
>> or when forced at gunpoint by government for safety reasons.
>> For fundamental research usually most computational resources get
>> thrown into battle, which is for nearly 100% sponsored
>> by government be it directly or indirectly.
>> In fact there is entire industries, especially clean energy
>> research, that is 100% paid by government subsidies.
>> Investigations after new products, in 99% of cases get funded by
>> government; usually the name of a big company is behind it,
>> but it's still government dudes who sit in that company doing the
>> research. What matters of course is who pays it.
>> The exception is persons who have their own company and are some
>> sort of big expert at a certain area. Take me.
>> I want to do some research after learning algorithms, at a very
>> high level (so far they didn't even conclude the right
>> value for material for chess, let alone that they were useful at
>> anything other than debugging your code, which is a big
>> important thing by the way).
>> Without getting the help of someone working at an university to
>> run such a thing overnight at a 160-200 cores,
>> i would of course only be able to run it at a single pc. Now i
>> don't get any money for all this from government,
>> so the big question is whether i'm supported by government then or
>> not.
>> What i do know is that all public research here at government
>> level is major shit. This is explainable by the fact that a field
>> is nonstop
>> on the move, only those who have been busy in a field for 10+
>> years have any clue what happens there and are the first to try
>> something;
>> the PHD's are simply busy with inferior software meanwhile having
>> massive hardware that's idle. A big problem is the nonpublishing of
>> accomplishments; First of all i'm not publishing *how* i'm doing
>> the learning. When some university offers me a job, i might
>> publish things,
>> i'm not getting paid for publications. Why help "the competition"?
>> What might get published is the guy who helps me; one tiny thing
>> of all this he'll publish probably. That's maybe 1/100 of the
>> conclusions
>> drawn.
>> Only the NSA type organisations know as they all spy on the
>> internet, and you BET that all big
>> countries know exactly what i'm doing, they all tap the internet
>> like crazy there.
>> If 1 researcher is real brilliant and CAN need some big
>> computation, there is at least a 100 spies who do know something
>> of that field,
>> and have to interpret the tapped data. What i do not know, as i
>> don't work for any of the N*SA type organisations, is in how far
>> *those*
>> verify things using massive hardware.
>> Most researchers have no clue how big the spying and anti-spying
>> is; most might get total scared if they'd realized how much they
>> get protected.
>> It is easier to steal it than to find an Einstein in your own
>> nation figuring it out.
>> That is the principle that every nation uses; CIA is a very tiny
>> organisation compared to what other nations have,
>> note that the weird thing is that CIA is notorious for violating
>> agreements with other nations (not spying in friendly nations,
>> to give an example; and also passing on information to their
>> companies that they got from the information stream they
>> got from friendly nations spying onto their own companies).
>> The fact that i write this down already means i have not been
>> employed in that field nor am; otherwise i would not even
>> *mention* the word.
>> Or to quote someone who works in that field when i said that there
>> gotta be an awful lot of civil servants in Netherlands busy in
>> that field,
>> as there is 3+ million tourists a year in Netherlands:
>> "we have tourism in Spain also".
>> Yet nothing on the internet is safe, that's the basic problem
>> here. It's too easy to hack internet lines. 1024 bits or 2048 RSA
>> or something that
>> gets used for SSH?
>> 1024 bits is too easy to crack for 1 organisation of each country
>> by simply throwing specialized hardware at it.
>> 2048 bits RSA is a tad harder, but also not a problem.
>> 128 bits AES?
>> No manner to attack it is public known (would only deliver $20k,
>> which is too little anyway, for the risk you take publishing a
>> method).
>> Yet even if there would be some sort of randomized GA-algorithm
>> that needs some sort of Pollard-Rho order O( 2 ^ 0.25n ),
>> then a simple dedicated cpu can already crack it handsdown.
>> Obviously most companies are not busy spamming the net what they
>> do with clusters or do not do.
>> Keeping it secret for their competitors is just too important, yet
>> if you ask me i feel big companies do real little
>> research in areas that do not lead directly to products of them.
>> There might be 1 or 2 exceptions, like oil companies,
>> but the question is in how far researchers there can be seen as
>> employees of that company as they get indirectly
>> sponsored by government whose interest in everything that happens
>> with oil is real big.
>> If you ask me however, way too little fundamental research gets
>> sponsored by government.
>> If you see just in a tiny nation like netherlands; the number of
>> 'researchers' that work for government is just a very small part of
>> the number of professors. It's like 1 researcher for each 2
>> professors; PHD's not counted.
>> Note this is also what a few studies recently showed.
>> Just the comparision to intelligence agencies which can put into
>> action each one of them quarter of a million to millions of people,
>> and who hire real easy people be it direct or indirect, it is
>> obvious that the saying: "Better well stolen than bad invented",
>> gets written in just too big capitals.
>> As we're speaking of a 1000 individuals in a nation having 16.5
>> million inhabitants, and only very few of those have
>> time to devote a big part of their time to do research; maybe 5% is?
>> Most are too busy with meetings and other organisational work and
>> posting on the internet.
>> Realize that same nation also has over a 1000 companies with
>> more than 1000 employees and half a million charity organisations.
>> That really lets fundamental research look like something that
>> gets total neglected.
>> Vincent
>> On Oct 7, 2008, at 9:34 AM,
>> wrote:
>>> Andrea,
>>>
>>> MPI is of course used by many applications running on commercial
>>> clusters.
>>> Two obvious examples are computational chemistry by the drug
>>> companies
>>> and computational fluid dynamics (CFD) for aerospace companies
>>> and F1 design teams.
>>>
>>> These are all long-term 'traditional' uses of MPI for scientific/
>>> engineering codes.
>>>
>>> Is this what you are asking? Or are you thinking of non-
>>> traditional uses in say computational finance or gaming sites?
>>>
>>> Daniel
>>>
>>> -------------------------------------------------------------
>>> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
>>> One Bridewell St., Mobile: +44 (0)779 209 1851
>>> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
>>> ----------------------- www.quadrics.com --------------------
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: beowulf-bounces at beowulf.org [mailto:beowulf-
>>> bounces at beowulf.org] On Behalf Of Andrea Di Blas
>>> Sent: 13 August 2008 00:37
>>> To: beowulf at beowulf.org
>>> Subject: [Beowulf] large MPI adopters
>>>
>>> hello,
>>>
>>>
>>> I am curious about what companies, besides the national labs of
>>> course,
>>> use any implementation of MPI to support large applications of
>>> any kind,
>>> whether only internally (like mapreduce for google, for example)
>>> or not.
>>>
>>> does anybody know of any cases?
>>> thank you and best regards,
>>>
>>>
>>> andrea
>>>
>>>
>>>
>>>
>>> --
>>> Andrea Di Blas, UCSC
>>> School of Engineering
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> Gerry Creager -- gerry.creager at tamu.edu
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Tue Oct 7 11:45:52 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Tue, 07 Oct 2008 11:45:52 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
Message-ID: <48EB8430.2070102@neuralbs.com>
I've read the entire thread up to now and noticed no one mentioned the
parallel bzip2 (pbzip2 http://compression.ca/pbzip2/) utility. Although
not strictly compatible with bzip2 when compressing large files, it's
still a valid compressor imho. More below...
Xu, Jerry wrote:
> Hello,
>
> Currently I generate nearly one TB data every few days and I need to pass it
> along enterprise network to the storage center attached to my HPC system, I am
> thinking about compressing it (most tiff format image data) as much as I can, as
> fast as I can before I send it crossing network ...
I performed a test on *uncompressed* tif images (RAW images from a Canon
EOS camera iirc) of many 32MB images tarred into a single 2.9Gig file
put into /dev/shm (simulating stream). Here is the time characteristics
for compressing the 2.9gig file using pbzip2:
kyron at kyron ~ $ /usr/bin/time -v pbzip2 -b15 -k /dev/shm/TEST.tar -c
>/dev/shm/TEST.tar.bzip
Command being timed: "pbzip2 -b15 -k /dev/shm/TEST.tar -c"
User time (seconds): 415.95
System time (seconds): 4.89
Percent of CPU this job got: 394%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:46.56
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 677802
Voluntary context switches: 4196
Involuntary context switches: 57332
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
The system has an Intel Quad Core Q6700 with 8Gigs of RAM (forced at
667MHz due to Asus's P35 implementation limitations).
Making sense of the figures, the file is 3046748160 bytes, the output
file is 1427448276 bytes (2 to 1 ratio...which will depend on image
contents of course).
It took 106.56 seconds (1:46.56 seconds / 4 cores) to process the
3046748160 which translates into a processing speed of ~ 28E6 bytes/sec
processing speed. Modern HDDs and GigE surpass this processing speed
comfortably.
Since it's been brought up in the thread, here are some figures for a
binary database (float vectors of pattern recognition
characteristics)...and note how compressible the file ends up being:
kyron at kyron ~ $ /usr/bin/time -v pbzip2 -b15 -k
~/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c
>/dev/shm/TEST.tar.bzip
Command being timed: "pbzip2 -b15 -k
/home/kyron/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c"
User time (seconds): 1096.09
System time (seconds): 7.12
Percent of CPU this job got: 395%
Elapsed (wall clock) time (h:mm:ss or m:ss): 4:38.65
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 135624
Voluntary context switches: 10233
Involuntary context switches: 154397
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
featg_row.dat if 1454853604 bytes and compresses down to 238M (guess
there is lots of redundancy in the features ;P ) But still, 1454853604
bytes / 278.65 seconds = 5.2E6 bytes/sec processing speed; still
insufficient compared to current hardware.
> So, I am wondering whether
> anyone is familiar with any hardware based accelerator, which can dramatically
> improve the compressing procedure.. suggestion for any file system architecture
> will be appreciated too.. I have couple of contacts from some vendors but not
> sure whether it works as I expected, so if anyone has experience about it and
> want to share, it will be really appreciated !
>
> Thanks,
>
> Jerry
>
> Jerry Xu PhD
> HPC Scientific Computing Specialist
> Enterprise Research Infrastructure Systems (ERIS)
> Partners Healthcare, Harvard Medical School
> http://www.partners.org
>
> The information transmitted in this electronic communication is intended only
> for the person or entity to whom it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or other
> use of or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received this
> information in error, please contact the Compliance HelpLine at 800-856-1983 and
> properly dispose of this information.
>
>
>
HTH
Eric Thibodeau
PS: Interesting figures, I couldn't resist compressing the same binary
DB on a 16Core Opteron (Tyan VX50) machine and was dumbfounded to get
horrible results given the same context. The processing speed only came
up to 6.4E6 bytes/sec ...for 16 cores, and they were all at 100% during
the entire run (FWIW, I tried different block sizes and it does have an
impact but this also changes the problem parameters).
eric at einstein ~ $ /usr/bin/time -v pbzip2 -b15 -k
~/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c
>/dev/shm/TEST.tar.bzip
Command being timed: "pbzip2 -b15 -k
/export/livia/home/parallel/eric/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat
-c"
User time (seconds): 3356.42
System time (seconds): 5.20
Percent of CPU this job got: 1481%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:46.94
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 345519
Voluntary context switches: 4088
Involuntary context switches: 9036
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Tue Oct 7 14:07:14 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Tue, 07 Oct 2008 14:07:14 -0400
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To:
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48EB8430.2070102@neuralbs.com>
Message-ID: <48EBA552.60305@neuralbs.com>
Dmitri Chubarov wrote:
> Hello,
>
> we have got a VX50 down here as well. We have observed very different
> scalability on different applications. With an OpenMP molecular
> dynamics code we have got over 14 times speedup while on a 2D finite
> difference scheme I could not get far beyond 3 fold.
>
2D finite difference can be comm intensive is the mesh is too small for
each processor to have a fair amount of work to do before needing the
neighboring values from a "far" node.
> On Tue, Oct 7, 2008 at 10:45 PM, Eric Thibodeau wrote:
>
>> PS: Interesting figures, I couldn't resist compressing the same binary DB on
>> a 16Core Opteron (Tyan VX50) machine and was dumbfounded to get horrible
>> results given the same context. The processing speed only came up to 6.4E6
>> bytes/sec ...for 16 cores, and they were all at 100% during the entire run
>> (FWIW, I tried different block sizes and it does have an impact but this
>> also changes the problem parameters).
>>
>
> Reading your message in the Beowulf list I should say that it looks
> interesting and probably shows something happening with the memory
> access on the NUMA nodes. Did you try to run the archiver with
> different affinity settings?
>
I don't have affinity control over the app per say. I would have to
look/modify pbzip's code. Although, note that the PID's assignment to
one processor is governed by the kernel and is thus a scheduler issue.
Also note that I have noticed that the kernel doesn't just have fun
moving the processes around the cores.
> We have observed that the memory architecture shows some strange
> behaviour. For instance the latency for a read from the same NUMA node
> for different nodes varies significantly.
>
This is the nature of NUMA. Furthermore, if you have to cross to a far
CPU, the latency is also dependent on the CPU's load.
> Also on the profiler I often see that x86 instructions that have one
> of the operands in memory may
> take disproportionally long. I believe that could explain the 100% CPU
> load reported by the kernel.
>
How do you identify the specific instruction using a profiler, this is
something that interests me.
> From the very little knowledge of this platform that we have got, I
> tend to advise the users not to expect good speedup on their
> multithreaded applications.
Using OpenMP (from GCC 4.3.x) and an embarrassingly parallel problem
(computing K-Means on a large database), I do get significant speedup
(15-16).
> Yet it would be interesting to get a
> better understanding of the programming techniques for this sedecimus
> and the similar machines.
OpenMP is IMHO the easiest one that will bring you the most performance
out of 3 lines of #pragma directives. If you manage to get a cluster of
VX50s, then learn a bit of MPI to glue all of this together ;)
> Even more so due to the QPI systems becoming
> commercially available very soon.
Don't know that one (QPI)...oh...new Intel stuff...no matter how much I
try to stay ahead, I'm always years behind!
> At the moment we have got a few
> small kernels written in C and Fortran with OpenMP that we use to
> evaluate different parallelization strategies. Unfortunately, there
> are no tools I would know of that could help to explain what's going
> on inside the memory of this machine.
>
Of course, check out TAU (
http://www.cs.uoregon.edu/research/tau/home.php ), it will at least help
you identify bottlenecks and give you an impressive profiling
infrastructure.
> I am very much interested to hear more about your experience with VX50.
>
> Best regards,
> Dima Chubarov
>
> --
> Dmitri Chubarov
> junior researcher
> Siberian Branch of the Russian Academy of Sciences
> Institute of Computational Technologies
> http://www.ict.nsc.ru/indexen.php
>
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From dmitri.chubarov at gmail.com Tue Oct 7 14:34:41 2008
From: dmitri.chubarov at gmail.com (Dmitri Chubarov)
Date: Wed, 8 Oct 2008 01:34:41 +0700
Subject: [Beowulf] Accelerator for data compressing
In-Reply-To: <48EBA552.60305@neuralbs.com>
References: <200810021900.m92J05RP012792@bluewest.scyld.com>
<552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org>
<48EB8430.2070102@neuralbs.com>
<48EBA552.60305@neuralbs.com>
Message-ID:
>
> 2D finite difference can be comm intensive is the mesh is too small for each
> processor to have a fair amount of work to do before needing the neighboring
> values from a "far" node.
>
Actually it seems that with VX50 the same node may be the "far" node.
At least that's what I see
from the NUMA Analysis test from TAU Wiki:
http://www.nic.uoregon.edu/tau-wiki/Guide:Opteron_NUMA_Analysis
Do not have the numbers at hand but that was the impression.
>
> How do you identify the specific instruction using a profiler, this is
> something that interests me.
>
I am using the Performance Analyzer that comes with Sun Studio 12. It
provides a per instruction profile view of the disassembly.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From gerry.creager at tamu.edu Wed Oct 8 08:00:36 2008
From: gerry.creager at tamu.edu (Gerry Creager)
Date: Wed, 08 Oct 2008 07:00:36 -0500
Subject: [Beowulf] large MPI adopters
In-Reply-To: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com><3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
<6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org>
Message-ID: <48ECA0E4.4060000@tamu.edu>
Tom,
I looked at that and *THOUGHT* it looked funny, but I was unable to see
the typo. Yeah, OpenMP. Both have their place, which is sometimes
integrated into the same application!
gerry
Tom Elken wrote:
>> [mailto:beowulf-bounces at beowulf.org] On Behalf Of Gerry Creager
>> The list isn't small: As Dan mentions, the drug companies are
>> big users,
>> although there's a fair bit of openMPI operation in their codes, too.
> Gerry,
> I'm sure you meant
> OpenMP
> (standard shared-memory threaded parallelism)
>
> So easy for fingers to type "MPI".
>
> Even though MPI has far more usage than OpenMP, I have to stick up for
> it as a former shared-memory advocate from my SGI days and as a member
> of the OpenMP ARB.
>
> -Tom
>
>> CFD in aerospace, and auto manufacturers. The US military almost
>> certainly does parallel processing involving both message passing and
>> Monte Carlo simulations. I've written code that solves physical and
>> satellite geodesy problems using PVM (but I was young and that was a
>> long time ago; today I'd use MPI); a variant of that was apparently
>> moved to the production world after I changed positions and couldn't
>> support it anymore.
>>
>> I'm not sure to tell you who the biggest corporate users are.
>> It likely
>> depends on one's exposure to even have an opinion. However, for the
>> biggest corporate users I'm aware of in my immediate area,
>> Schlumberger,
>> Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
>> obviously a very incomplete list.
>>
>>
>> Gerry
>>
>
>>> From: beowulf-bounces at beowulf.org
>> [mailto:beowulf-bounces at beowulf.org]
>>>> On Behalf Of Andrea Di Blas
>>>> Sent: 13 August 2008 00:37
>>>> To: beowulf at beowulf.org
>>>> Subject: [Beowulf] large MPI adopters
>>>>
>>>> hello,
>>>>
>>>>
>>>> I am curious about what companies, besides the national
>> labs of course,
>>>> use any implementation of MPI to support large
>> applications of any kind,
>>>> whether only internally (like mapreduce for google, for
>> example) or not.
>>>> does anybody know of any cases?
>>>> thank you and best regards,
>>>>
>>>>
>>>> andrea
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrea Di Blas, UCSC
>>>> School of Engineering
>>>> _______________________________________________
>>>> Beowulf mailing list, Beowulf at beowulf.org
>>>> To change your subscription (digest mode or unsubscribe) visit
>>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>>
>>>> _______________________________________________
>>>> Beowulf mailing list, Beowulf at beowulf.org
>>>> To change your subscription (digest mode or unsubscribe) visit
>>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>> --
>> Gerry Creager -- gerry.creager at tamu.edu
>> Texas Mesonet -- AATLT, Texas A&M University
>> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
>> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe)
>> visit http://www.beowulf.org/mailman/listinfo/beowulf
>>
--
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 8 10:09:36 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Wed, 8 Oct 2008 16:09:36 +0200 (CEST)
Subject: [Beowulf] large MPI adopters
In-Reply-To:
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
<0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
Message-ID:
On Tue, 7 Oct 2008, Vincent Diepeveen wrote:
> Drug type companies eat typical 0.5% system time.
Could you please explain what you mean here ? That they buy lots of
CPUs but only use a fraction of them or only for a fraction of time ?
> They are far behind, which is not so logical; i would consider it
> old fashioned to do experiments on monkeys (Bar-ilan brain
> experiments on monkeys rings a bell?), so doing everything in
> software is far more logical.
Maybe your knowledge about the pharma industry is far behind ? "Doing
everything in software is far more logical" only when you can exactly
express biological processes in mathematical terms which is not true
in the great majority of cases. F.e. molecular dynamics (often used to
simulate interactions at molecular level, f.e. of a drug injected into
the body) is based on several approximations and the implementations
are subject to numerical errors. Would you take a drug that was
produced based only on a simulation with molecular dynamics and found
to be good enough ?
Similar concerns apply to all levels - the brain, as you mention it,
is still very much an enigma, otherwise artificial intelligence would
be all around us (or not, if deemed to dangerous...). And when you
don't know how an organ or system works, how can you develops drugs
only by simulations ?
> When someone is really in big need for big calculation power, one
> makes dedicated cpu's for it.
I tend to agree, provided that the dedicated CPU brings a really big
advantage, like an order of more of magnitude. F.e., to stay in the
pharma related field, that's the bussiness case for DEShaw Research's
molecular dynamics chip (Anton) which is supposed to bring a
significant increase in speed for MD simulations compared with
software solutions running on general purpose CPUs (or GPUs).
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From diep at xs4all.nl Wed Oct 8 12:32:42 2008
From: diep at xs4all.nl (Vincent Diepeveen)
Date: Wed, 8 Oct 2008 18:32:42 +0200
Subject: [Beowulf] large MPI adopters
In-Reply-To:
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
<0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
Message-ID: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl>
On Oct 8, 2008, at 4:09 PM, Bogdan Costescu wrote:
> On Tue, 7 Oct 2008, Vincent Diepeveen wrote:
>
>> Drug type companies eat typical 0.5% system time.
>
> Could you please explain what you mean here ? That they buy lots of
> CPUs but only use a fraction of them or only for a fraction of time ?
>
When you add up others.
Note i see in IWR you got a 160 node opteron cluster, what software
eats system time on that cluster?
There must be graphs about it, want to share one? Also interesting to
know is how many cores most jobs eat!
Note that i left out 1 important factor which is not a factor in
every nation: that's the amount of system time airindustry eats of
supercomputers
designing new stuff. It is very significant together with car
manufacturers. With 160 nodes machine, even if that would be power6
nodes,
you can't even design a wing for aircraft.
>> They are far behind, which is not so logical; i would consider it
>> old fashioned to do experiments on monkeys (Bar-ilan brain
>> experiments on monkeys rings a bell?), so doing everything in
>> software is far more logical.
>
> Maybe your knowledge about the pharma industry is far behind ?
> "Doing everything in software is far more logical" only when you
> can exactly express biological processes in mathematical terms
> which is not true in the great majority of cases. F.e. molecular
> dynamics (often used to simulate interactions at molecular level,
> f.e. of a drug injected into the body) is based on several
> approximations and the implementations are subject to numerical
> errors. Would you take a drug that was produced based only on a
> simulation with molecular dynamics and found to be good enough ?
>
Well, i sure avoid hospital visits, you?
As you know the healthcare commission is basically old men who spread
careful chosen sentences in such a manner that
everything looks fine on this planet and that there is nothing to
worry about; last time i spoke with them on EMF's a guy being spokesman
on this subject, he didn't even know how the electricity system in
his own house worked (whether on the first floor of his house
he had earth).
So lucky to introduce to a new drug, there is some burocratic
systems. There is several phases of testing that a drug
has to pass in order to get accepted as a drug that is legal. Not to
confuse with the 'advice' on whether a drug gets prescribed;
they want way cheaper drugs for that usually than the ones that might
maybe be better (say 0.5% better for a lot of money extra,
but if you're that guy that didn't get cured as a result of that,
your family will regret it bigtime).
Computing power or not, most manufacturers simply want to sell a
product, so the amount of systemtime invested on a computer,
in order to sell, is total irrelevant to them to get a drug accepted
usually. The first and most important part of the show to get a drug
accepted is getting invited to the table to show your drug. Computing
power is nice to mention here when that is usually not a factor
at all to get a new medicine accepted.
But it may change, let's hope so...
Maybe a new breed of company can do the job. A breed where making
profit is not most important :)
> Similar concerns apply to all levels - the brain, as you mention
> it, is still very much an enigma, otherwise artificial intelligence
> would be all around us (or not, if deemed to dangerous...). And
> when you don't know how an organ or system works, how can you
> develops drugs only by simulations ?
>
>> When someone is really in big need for big calculation power, one
>> makes dedicated cpu's for it.
>
> I tend to agree, provided that the dedicated CPU brings a really
> big advantage, like an order of more of magnitude. F.e., to stay in
> the pharma related field, that's the bussiness case for DEShaw
> Research's molecular dynamics chip (Anton) which is supposed to
> bring a significant increase in speed for MD simulations compared
> with software solutions running on general purpose CPUs (or GPUs).
Some hardware guys can better comment here. Making your own FPGA
chippie is rather cheap.
Finding and paying a few good guys to get the chip done is toughest.
You basically need 1 good
programmer, the rest can get hired parttime in fact. After chip works,
to print a 1000 cards, it is like a 100 dollar for pci-card and 50
dollar a chip.
So $150k for computing power that total annihilates any supercomputer
of now and the NEAR future for your specific application.
Real cpu's are a lot faster of course, price for them i do not have
at hand. I heard it could be cheaper
than this in fact if you regurarly print a batch. Such a batch is
producing less cpu's than the FPGA, where you pay for 1 cpu a constant
amount. I tend to remember that good cpu designers, not to mention
memory controller guys, are rather expensive folks, as
they are very popular; on some other mailing list with some very
funny lastnames, i see nonstop job offers for that type of folks.
Vincent
> --
> Bogdan Costescu
>
> IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
> Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
> E-mail: bogdan.costescu at iwr.uni-heidelberg.de
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 8 14:00:19 2008
From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu)
Date: Wed, 8 Oct 2008 20:00:19 +0200 (CEST)
Subject: [Beowulf] large MPI adopters
In-Reply-To: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
<0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
<57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl>
Message-ID:
[ Trimmed CC list ]
On Wed, 8 Oct 2008, Vincent Diepeveen wrote:
> When you add up others.
It's interesting that you make this statement along with another one
saying that many companies that have clusters or other parallel
computing resources don't make them public. So how can you arrive to
such a figure ???
> Note i see in IWR you got a 160 node opteron cluster, what software
> eats system time on that cluster?
In my vocabulary, system time is the time used by the system (kernel,
maybe daemons) as opposed to user time.
What probably interests you is what kind of software is running on
HELICS; if so, how come you managed to see the hardware page but
missed the projects list ? It's linked from the main HELICS page,
but to make it easier for you this is the direct link:
http://helics.iwr.uni-heidelberg.de/users/project_overview.php
The 160 nodes are only the newer part called HELICS II; and there are
also other clusters in IWR :-)
> There must be graphs about it, want to share one?
Measuring what ? Generally speaking, I don't produce graphs, text
output is enough for me to know that the cluster is busy :-)
> Also interesting to know is how many cores most jobs eat!
Most jobs running on HELICS II start 4 processes per node. There are 4
cores per node, so that's 1 process per core. Some of the software
(mostly developed in-house) requires a lot of memory per process, so
then less cores per node are being used - it's a tradeoff that was
factored in the design.
> that's the amount of system time airindustry eats of supercomputers
> designing new stuff. It is very significant together with car
> manufacturers.
And how do you know that ? :-)
> With 160 nodes machine, even if that would be power6 nodes, you
> can't even design a wing for aircraft.
Aircraft design as a research subject can probably easily earn
computer time on the many parallel computing resources in Germany that
any researcher working in a German university can apply for. Some of
the parallel computing resources are better suited for certain tasks
(like the NEC SX8 in Stuttgart for vector software), so we're free to
choose the one that fits best. Local resources are in some cases
preferred because they can be easier controlled (f.e. hardware choices
to make it fit for a particular application that is run very often) or
getting access to. So the size of the local cluster(s) is not a
limiting factor for the science that gets done.
Getting non-x86(_64) compatible CPUs (like Power-something) for HELICS
was considered but rejected because of the already-mentioned problem
of binary-only software from ISVs.
> Well, i sure avoid hospital visits, you?
I do go to the hospital to visit ill relatives or friends :-)
> As you know the healthcare commission is basically old men
You seem to have a lot of knowledge about everything, I don't - so I
can't comment on your statements. ;-)
> Computing power or not, most manufacturers simply want to sell a
> product, so the amount of systemtime invested on a computer, in
> order to sell, is total irrelevant to them to get a drug accepted
> usually.
Indeed, the computer is involved usually only in the initial stages,
when many compounds are screened and only very few go further.
Clinical trials are very expensive and time consuming, so the computer
time used initially is only a fraction of the total time needed to put
out a new drug. But this is not because the pharma companies would
want to keep it so long, it's because all these further tests are
supposed to keep the later users of these drugs out of trouble and to
provide that (sometimes very long...) list of side-effects they are
required to put in the box. It's not the companies who find the
computer time irrelevant, it's the whole system that makes it so.
> Maybe a new breed of company can do the job. A breed where making
> profit is not most important :)
"Imagine there's no money" (paraphrasing John Lennon) :-)
> Some hardware guys can better comment here. Making your own FPGA
> chippie is rather cheap. Finding and paying a few good guys to get
> the chip done is toughest. You basically need 1 good programmer, the
> rest can get hired parttime in fact.
Maybe you should read more about the Anton chip and the team that
created it. That would better fit into the 'lot of knowledge about
everything' figure that I mentioned above :-)
> So $150k for computing power that total annihilates any
> supercomputer of now and the NEAR future for your specific
> application.
Whew, do you wanna become partners so that we can sell this idea to
NASA ? :-)))
> I tend to remember that good cpu designers, not to mention memory
> controller guys, are rather expensive folks
Indeed. Then look at the Anton chip and see why your 1+peanuts
expert/project idea doesn't fit ;-)
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From csamuel at vpac.org Wed Oct 8 23:40:57 2008
From: csamuel at vpac.org (Chris Samuel)
Date: Thu, 9 Oct 2008 14:40:57 +1100 (EST)
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To:
Message-ID: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org>
----- "Igor Kozin" wrote:
> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
> http://helmer3.sfe.se/
Sadly it looks like that's based on the single precision
numbers for graphics cards, rather than double..
--
Christopher Samuel - (03) 9925 4751 - Systems Manager
The Victorian Partnership for Advanced Computing
P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Dan.Kidger at quadrics.com Thu Oct 9 08:08:22 2008
From: Dan.Kidger at quadrics.com (Dan.Kidger at quadrics.com)
Date: Thu, 9 Oct 2008 13:08:22 +0100
Subject: [Beowulf] Pretty High Performance Computing
In-Reply-To: <9f8092cc0809240805y4e8d56dsc08b0e3ce83fea45@mail.gmail.com>
References: <518096.44032.qm@web37908.mail.mud.yahoo.com>
<9f8092cc0809240805y4e8d56dsc08b0e3ce83fea45@mail.gmail.com>
Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A0493821198@quadbrsex1.quadrics.com>
John,
Yes indeed at that Daresbury Machine Evaluation Workshop - quite a while ago - maybe 2001 or 2002
Mike gave out as the freebie on their booth Boxer Shorts with the Streamline logo on.
Compared with giving away say t-shirts, I am not sure how much market exposure they got (!)
Daniel
-------------------------------------------------------------
Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
One Bridewell St., Mobile: +44 (0)779 209 1851
Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
----------------------- www.quadrics.com --------------------
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of John Hearns
Sent: 24 September 2008 16:06
To: beowulf at beowulf.org
Subject: Re: [Beowulf] Pretty High Performance Computing
2008/9/24 Ellis Wilson >
This assumes my understanding of middleware is correct in that it is a
package or entire system that simplifies things by being somewhat
blackboxed and ready to go. Anything canned like tuna is bound to
contain too much salt.
I believe that that Mike Rudgyard, the chief exec of Streamline Computing, coined the term 'underware' for this.
This wa s presentation at one of the Daresbury Lab's Machine Evaluation Workshops, which I can't find via Google at the moment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From Dan.Kidger at quadrics.com Thu Oct 9 09:18:54 2008
From: Dan.Kidger at quadrics.com (Dan.Kidger at quadrics.com)
Date: Thu, 9 Oct 2008 14:18:54 +0100
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org>
References:
<2045508708.1313971223523657435.JavaMail.root@mail.vpac.org>
Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com>
Looking at the CAD drawings - is there any reason why the nodes are built into circular columns, rather than traditional cuboid boxes?
If anything such a design does not make full use of the physical space available.
And if you did pack the columns as drawn (with vertical water pipes in the gaps) - how do you service the nodes to replace failed components, water leaks et al. ?
And also (I do work for an interconnect company after all) how would all this get wired up together as a cluster?
Daniel
-------------------------------------------------------------
Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
One Bridewell St., Mobile: +44 (0)779 209 1851
Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
----------------------- www.quadrics.com --------------------
-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Chris Samuel
Sent: 09 October 2008 04:41
To: Beowulf List
Subject: Re: [Beowulf] FY;) the Helmer cluster
----- "Igor Kozin" wrote:
> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
> http://helmer3.sfe.se/
Sadly it looks like that's based on the single precision
numbers for graphics cards, rather than double..
--
Christopher Samuel - (03) 9925 4751 - Systems Manager
The Victorian Partnership for Advanced Computing
P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 9 10:37:30 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 9 Oct 2008 15:37:30 +0100
Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC
Message-ID: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com>
As a Guardian reader, I have been reading the Thursday Technology supplement
for many years.
On the train this morning, I opened it to find Jack Schofield has an article
on Windows HPC and the
Cray deskside supercomputer.
http://www.guardian.co.uk/technology/2008/oct/09/computing.microsoft
Has Jack been reading the Beowulf list?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From james.p.lux at jpl.nasa.gov Thu Oct 9 11:45:07 2008
From: james.p.lux at jpl.nasa.gov (Lux, James P)
Date: Thu, 9 Oct 2008 08:45:07 -0700
Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC
In-Reply-To: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com>
References: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com>
Message-ID:
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of John Hearns
Sent: Thursday, October 09, 2008 7:38 AM
To: beowulf at beowulf.org
Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC
As a Guardian reader, I have been reading the Thursday Technology supplement for many years.
On the train this morning, I opened it to find Jack Schofield has an article on Windows HPC and the
Cray deskside supercomputer.
http://www.guardian.co.uk/technology/2008/oct/09/computing.microsoft
Has Jack been reading the Beowulf list?
---
>From the article:
"For example, Faenov says Excel 2007 supports "transparent parallelism", so that an HPC machine can speed up everyday workflows in financial institutions where they have spreadsheets that take hours or even days to run. (Crazy, I know.)"
No doubt, those models were used to do interesting things like calculate expected risk of default for the multiple tranches in various securitized debt obligations. I'm not sure that, philosopically, more speed is good in this area. It's one thing for a mathematical proof to be so big only a computer can do it (e.g. 4color map theorem), or when you want to run numerical models of weather or fluid dynamics. At least in those areas, there's a fair amount of work that goes into validation of the underlying numerics.
Excel, on the other hand, is probably not the best tool in the world for hard core numerical analysis (what with *interesting* phenomena like the famous 850*77.1= 100000 bug, ... Yes it was fixed, but it's not like there's some rigorous verification of Excel's computations that's available for inspection)http://www.lomont.org/Math/Papers/2007/Excel2007/Excel2007Bug.pdf has an explanation
And, as it happens this particular HPC application space (long running spreadsheet calcs) does exist. Many years ago, my sister had a job at a bank where part of the task was running Lotus 1-2-3 worksheets on a PC that took hours to recalc on a PC/AT. I actually contemplated designing and building an ECL IBM PC emulator for this market, but then Intel came out with the 386, etc.. Much better system solution to solve it in silicon.
Jim
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Thu Oct 9 11:49:08 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Thu, 9 Oct 2008 16:49:08 +0100
Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC
In-Reply-To:
References: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com>
Message-ID: <9f8092cc0810090849o4bc48d46i76029a853440905f@mail.gmail.com>
2008/10/9 Lux, James P
>
>
>
> And, as it happens this particular HPC application space (long running
> spreadsheet calcs) does exist. Many years ago, my sister had a job at a
> bank where part of the task was running Lotus 1-2-3 worksheets on a PC that
> took hours to recalc on a PC/AT.
Indeed. I have seen job adverts for City of London financial hoouses where
the job description was to write Excel macros and run spreadsheets for
traders. I'm somewhat glad I didn't apply - could be a tad stressful even
without the current downward trend.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From landman at scalableinformatics.com Thu Oct 9 13:59:48 2008
From: landman at scalableinformatics.com (Joe Landman)
Date: Thu, 09 Oct 2008 13:59:48 -0400
Subject: [Beowulf] OT: OhioLinuxFest this weekend
Message-ID: <48EE4694.4070108@scalableinformatics.com>
Hi folks
In case any beowulfers will be round the Columbus OH area this
Saturday, OhioLinuxFest (www.ohiolinux.org) will be on. We (Scalable)
will be there at a booth with some nVidia goodness in a deskside unit,
and something else as well (big evil grin: ΔV). Andrew Latham (a
lurker on this list, and all around good guy) will be there as well.
Stop by and say hi if you do attend. If enough people show up, we
may try for a Beowulf/Cluster/Storage BOF.
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Thu Oct 9 14:57:39 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Thu, 9 Oct 2008 13:57:39 -0500
Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly
crashing nodes.
Message-ID:
On my Centos system I installed kexec/kdump to investigate the cause of
some random system-crashes by getting access to a crash-dump. I installed
the rpm for kexec and then made the change to grub.conf that reserves the
additional memory for the new kernel.
Also configured kdump.conf. I start the kexec service.and then I tried to
simulate a crash by echo c to sysrq-trigger.
The system does crash and then after a while reboots itself. But I see no
vmcore when it coms back up. /var/crash is empty. This is when I tried to
write to local drive.
I also tried a nfs write but then still no success.
Any idea what could be missing in my steps? Or any other debug
suggestions? Any other kdump users on Beowulf?
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Thu Oct 9 15:19:20 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Thu, 9 Oct 2008 14:19:20 -0500
Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly
crashing nodes.
In-Reply-To: <48EE567C.3010503@gmail.com>
References:
<48EE567C.3010503@gmail.com>
Message-ID:
Hi Paolo,
The funny thing is that the console remains blank. We have all these
systems connected to a KVM and the kvm shows the system as actually
disconnected post the crash.
That is what makes it so hard to debug. No screen output at all.
-Rahul
On Thu, Oct 9, 2008 at 2:07 PM, Paolo Supino wrote:
> Hi Rahul
>
> Did you try to redirect console to a serial port? If a system crashes
> and all console messages (including kernel) will be sent to the serial
> console that will keep displaying the messages it received until the
> system is power cycled ...
>
>
>
>
>
> --
> ttyl
> Paolo
>
>
>
> Rahul Nabar wrote:
>> On my Centos system I installed kexec/kdump to investigate the cause of
>> some random system-crashes by getting access to a crash-dump. I installed
>> the rpm for kexec and then made the change to grub.conf that reserves the
>> additional memory for the new kernel.
>>
>> Also configured kdump.conf. I start the kexec service.and then I tried to
>> simulate a crash by echo c to sysrq-trigger.
>>
>> The system does crash and then after a while reboots itself. But I see no
>> vmcore when it coms back up. /var/crash is empty. This is when I tried to
>> write to local drive.
>>
>> I also tried a nfs write but then still no success.
>>
>> Any idea what could be missing in my steps? Or any other debug
>> suggestions? Any other kdump users on Beowulf?
>>
>
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Thu Oct 9 15:20:52 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Thu, 9 Oct 2008 14:20:52 -0500
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
Message-ID:
I have a PowerEdge SC 1435 that has a strange problem. We bought about 23 of
these for a cluster and machines have been failing in a somewhat random manner
in a peculiar way:
(1) Screen is blank. Front blue indicator turns steady orange.
(2) Cannot get it to reboot by pressing (or keeping depressed) the power button
(3) only way to reboot is to cycle the power.
(4) After reboot machine works fine again , till after a few days same failure.
Ran the dset and diagnostic CD but nothing relevant.
Any tips what could be the faulty component? Or debug ideas? Right now I'm
totally lost! Hardware / software? CPU / Motherboard / Power supply?
Anoybody knows what exactly makes the indicator LED turn steady orange from its
normal blue state? This is not one of the 4 numbered LEDs but the one to their
right.
I posted this problem on a PowerEdge mailing list but haven't gotten
very far yet. Any suggestions are appreciated!
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Thu Oct 9 15:35:07 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Thu, 9 Oct 2008 12:35:07 -0700
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <20081009193507.GB2340@bx9.net>
On Thu, Oct 09, 2008 at 02:20:52PM -0500, Rahul Nabar wrote:
> (2) Cannot get it to reboot by pressing (or keeping depressed) the power button
That's a pretty unusual hang. I bet that the reason that you don't get
a kernel crash dump is that the kernel doesn't run long enough after
the problem happens to create one.
Probably your fastest solution is to swap parts until works. Tedious,
but...
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Thu Oct 9 16:17:18 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Thu, 9 Oct 2008 15:17:18 -0500
Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly
crashing nodes.
In-Reply-To: <48EE6294.5050509@gmail.com>
References:
<48EE567C.3010503@gmail.com>
<48EE6294.5050509@gmail.com>
Message-ID:
On Thu, Oct 9, 2008 at 2:59 PM, Paolo Supino wrote:
> Hi Raul
>
> Can it be that console messages aren't being sent to VTY (standard PC
> monitor), but some other output line?
That makes sense. I'll try connecting a dumb monitor to the machine
and see what it reports.
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From paolo.supino at gmail.com Thu Oct 9 15:07:40 2008
From: paolo.supino at gmail.com (Paolo Supino)
Date: Thu, 09 Oct 2008 21:07:40 +0200
Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly
crashing nodes.
In-Reply-To:
References:
Message-ID: <48EE567C.3010503@gmail.com>
Hi Rahul
Did you try to redirect console to a serial port? If a system crashes
and all console messages (including kernel) will be sent to the serial
console that will keep displaying the messages it received until the
system is power cycled ...
--
ttyl
Paolo
Rahul Nabar wrote:
> On my Centos system I installed kexec/kdump to investigate the cause of
> some random system-crashes by getting access to a crash-dump. I installed
> the rpm for kexec and then made the change to grub.conf that reserves the
> additional memory for the new kernel.
>
> Also configured kdump.conf. I start the kexec service.and then I tried to
> simulate a crash by echo c to sysrq-trigger.
>
> The system does crash and then after a while reboots itself. But I see no
> vmcore when it coms back up. /var/crash is empty. This is when I tried to
> write to local drive.
>
> I also tried a nfs write but then still no success.
>
> Any idea what could be missing in my steps? Or any other debug
> suggestions? Any other kdump users on Beowulf?
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rlinesseagate at gmail.com Thu Oct 9 16:18:09 2008
From: rlinesseagate at gmail.com (Rob Lines)
Date: Thu, 9 Oct 2008 16:18:09 -0400
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <69592cde0810091318w70e8507ax4e2dae0aa849fbb5@mail.gmail.com>
On Thu, Oct 9, 2008 at 3:20 PM, Rahul Nabar wrote:
> I have a PowerEdge SC 1435 that has a strange problem. We bought about 23 of
> these for a cluster and machines have been failing in a somewhat random manner
> in a peculiar way:
>
> (1) Screen is blank. Front blue indicator turns steady orange.
>
> (2) Cannot get it to reboot by pressing (or keeping depressed) the power button
>
> (3) only way to reboot is to cycle the power.
>
> (4) After reboot machine works fine again , till after a few days same failure.
>
> Ran the dset and diagnostic CD but nothing relevant.
>
> Any tips what could be the faulty component? Or debug ideas? Right now I'm
> totally lost! Hardware / software? CPU / Motherboard / Power supply?
>
Have you checked in the baseboard management log to see if it is
throwing an error. Also check on the temperature of the machines. We
have had some pretty wierd issues with ram and CPU quirkyness when
they reach a high internal temperature. If you can do some poling
using ipmi on the nodes to record the current temp and fan data over
time so that you could see what it was at just before a crash you
might be able to point it to an environmental situation.
Hope this helps,
Rob
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From paolo.supino at gmail.com Thu Oct 9 15:59:16 2008
From: paolo.supino at gmail.com (Paolo Supino)
Date: Thu, 09 Oct 2008 21:59:16 +0200
Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly
crashing nodes.
In-Reply-To:
References:
<48EE567C.3010503@gmail.com>
Message-ID: <48EE6294.5050509@gmail.com>
Hi Raul
Can it be that console messages aren't being sent to VTY (standard PC
monitor), but some other output line?
I don't know what KVM you're using, but most KVMs keep an open line
only to the server they are displaying at the moment (and will send back
a signal if they get a probe from an another system they are connected
to) and won't buffer display output from the other systems they are
connected to. What I suggested is to connect a real serial terminal to
the Linux system. Make sure that all console messages (including kernel)
are sent to the serial console (the default is to send them to VTY which
is the standard display in a PC. If you can't connect a real serial
terminal than the closest thing to it would be a PC running putty (or
any other terminal emulation software) listening on COM1 and is
connected to the serial port of the server.
--
ttyl
Paolo
Rahul Nabar wrote:
> Hi Paolo,
>
> The funny thing is that the console remains blank. We have all these
> systems connected to a KVM and the kvm shows the system as actually
> disconnected post the crash.
>
> That is what makes it so hard to debug. No screen output at all.
>
> -Rahul
>
> On Thu, Oct 9, 2008 at 2:07 PM, Paolo Supino wrote:
>> Hi Rahul
>>
>> Did you try to redirect console to a serial port? If a system crashes
>> and all console messages (including kernel) will be sent to the serial
>> console that will keep displaying the messages it received until the
>> system is power cycled ...
>>
>>
>>
>>
>>
>> --
>> ttyl
>> Paolo
>>
>>
>>
>> Rahul Nabar wrote:
>>> On my Centos system I installed kexec/kdump to investigate the cause of
>>> some random system-crashes by getting access to a crash-dump. I installed
>>> the rpm for kexec and then made the change to grub.conf that reserves the
>>> additional memory for the new kernel.
>>>
>>> Also configured kdump.conf. I start the kexec service.and then I tried to
>>> simulate a crash by echo c to sysrq-trigger.
>>>
>>> The system does crash and then after a while reboots itself. But I see no
>>> vmcore when it coms back up. /var/crash is empty. This is when I tried to
>>> write to local drive.
>>>
>>> I also tried a nfs write but then still no success.
>>>
>>> Any idea what could be missing in my steps? Or any other debug
>>> suggestions? Any other kdump users on Beowulf?
>>>
>>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From csamuel at vpac.org Thu Oct 9 18:20:54 2008
From: csamuel at vpac.org (Chris Samuel)
Date: Fri, 10 Oct 2008 09:20:54 +1100 (EST)
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To: <831359607.1340021223590533957.JavaMail.root@mail.vpac.org>
Message-ID: <2098513302.1340111223590854452.JavaMail.root@mail.vpac.org>
----- "Rahul Nabar" wrote:
> (1) Screen is blank. Front blue indicator turns steady orange.
What we do on our CentOS cluster is disable the standard
screen blanker in our startup scripts with:
echo -e "\033[9;0]" >/dev/console
echo -e "\033[13]" >/dev/console
Very handy if a node wedges and won't respond to keyboard.
cheers,
Chris
--
Christopher Samuel - (03) 9925 4751 - Systems Manager
The Victorian Partnership for Advanced Computing
P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From kyron at neuralbs.com Thu Oct 9 22:52:22 2008
From: kyron at neuralbs.com (Eric Thibodeau)
Date: Thu, 09 Oct 2008 22:52:22 -0400
Subject: [Beowulf] FY;) the Helmer cluster
In-Reply-To: <0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com>
References: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org>
<0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com>
Message-ID: <48EEC366.7030201@neuralbs.com>
Dan.Kidger at quadrics.com wrote:
> Looking at the CAD drawings - is there any reason why the nodes are built into circular columns, rather than traditional cuboid boxes?
> If anything such a design does not make full use of the physical space available.
>
> And if you did pack the columns as drawn (with vertical water pipes in the gaps) - how do you service the nodes to replace failed components, water leaks et al. ?
>
> And also (I do work for an interconnect company after all) how would all this get wired up together as a cluster?
>
Awww com on, don't yo believe in fluffware? I say he goes full wifi :P
> Daniel
>
> -------------------------------------------------------------
> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
> One Bridewell St., Mobile: +44 (0)779 209 1851
> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
> ----------------------- www.quadrics.com --------------------
>
>
>
>
> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Chris Samuel
> Sent: 09 October 2008 04:41
> To: Beowulf List
> Subject: Re: [Beowulf] FY;) the Helmer cluster
>
>
> ----- "Igor Kozin" wrote:
>
>
>> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M
>> http://helmer3.sfe.se/
>>
>
> Sadly it looks like that's based on the single precision
> numbers for graphics cards, rather than double..
>
> --
> Christopher Samuel - (03) 9925 4751 - Systems Manager
> The Victorian Partnership for Advanced Computing
> P.O. Box 201, Carlton South, VIC 3053, Australia
> VPAC is a not-for-profit Registered Research Agency
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Fri Oct 10 01:54:03 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Fri, 10 Oct 2008 06:54:03 +0100
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com>
2008/10/9 Rahul Nabar
> I
>
> I posted this problem on a PowerEdge mailing list but haven't gotten
> very far yet. Any suggestions are appreciated!
>
> Yes.
(1) Tell your Dell salesman that you have asked for help on this problem on
a public mailing list for High Performance Computing. Tell him/her that you
need high level Dell support on this. There are Dell customers on this list.
(2) Suspect the RAM. Ask some serious questions of your Dell support about
RAM compatibility - HPC applications stress the RAM. Ask, and ask again, if
the specific RAM chips you have are certified for that motherboard. Use
dmidecode to read out the manufacturer codes of the RAM modules - do you
have a mix of manufacturers?
Ask and ask again about BIOS updates being available for these machines.
We had a case once of HP machines - even though the BIOSes were versioned
the same on 200 machines, there were some differences - turns out you had to
go as far as checking the build date.
Get the very latest BIOS version you can.
(3) The RAM will be the problem - but if you can keep notes and there are
specific machines which crash more than others point this out to Dell and
maybe suspect the PSUs being weak on those machines.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Fri Oct 10 04:17:27 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Fri, 10 Oct 2008 09:17:27 +0100
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To: <9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com>
References:
<9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com>
Message-ID: <9f8092cc0810100117u316b5f46j7d6630e6747c2d8f@mail.gmail.com>
2008/10/10 John Hearns
>
> (1) Tell your Dell salesman that you have asked for help on this problem
> on a public mailing list for High Performance Computing. Tell him/her that
> you need high level Dell support on this.
>
ps. By high level support I mean that you are put in direct contact with one
of the engineers responsible for the design of these systems in the factory.
You do not want to talk to the normal support chain, and be asked if you
have run some diagnostics program downloaded from the Dell site, etc.
As a general observation, not particularly aimed at Dell, I have seen this
type of behaviour before.
Beowulf clusters are built from COTS systems - servers which are disigned
for a workload of webserving, or running farms of virtual servers. I really
think that the manufacturers would be surprised at the typical HPC workload
- running all cores flat out at 100%, having applications which can
comfortably use a couple of hundred gigs of RAM.
My feeling is that the PSUs are specced to cope with these loads, but any
out of spec ones will be on the edge. RAM timings too - if you are stressing
all the RAM in a system you'll show up any weaknesses.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Fri Oct 10 10:36:53 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Fri, 10 Oct 2008 09:36:53 -0500
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
Message-ID:
>What we do on our CentOS cluster is disable the standard
>screen blanker in our startup scripts with:
>
>echo -e "\033[9;0]" >/dev/console
>echo -e "\033[13]" >/dev/console
>
>Very handy if a node wedges and won't respond to keyboard.
Great idea, Chris. I did not know this. That could be why I don't see
any output on my screens later.
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Fri Oct 10 10:47:32 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Fri, 10 Oct 2008 09:47:32 -0500
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
Message-ID:
>t's a pretty unusual hang. I bet that the reason that you don't get
>a kernel crash dump is that the kernel doesn't run long enough after
>the problem happens to create one.
Thanks Greg. I suspected that....I am actually curious: when exactly
can kdump be useful? If a crash is hardware precipitated the second
kernel never gets a chance to do what its supposed to. If it is
software related, and the first kernel actually has time to detect the
inconsistancy then it might as well "deal" with the offending process.
>Probably your fastest solution is to swap parts until works. Tedious,
>but...
That's exactly what I'm doing so far! :-) Problem is which ones? CPU /
MB/ Power supplies /RAM ? I've even received solutions as exotic as
re-flashing the BIOS / ESM firmware upgrades / processor reseating.
Any bets on the likelihood based on symptoms, intuition, and past
experience?
We've swapped CPUs and processors on all the offending nodes. Seems
to have worked so far (i.e. none of the swapped machines have
re-crashed) But I'm hesitant to conclude "problem solved" since all
this is only over the last 2 weeks.
I'm dreading the day when one of the swapped machines re-crashes!
Let's see........
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Fri Oct 10 10:54:35 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Fri, 10 Oct 2008 09:54:35 -0500
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
Message-ID:
>Have you checked in the baseboard management log to see if it is
>throwing an error.
Apparently the SC1435 does not have OpenManage. "Simple Computing" is
too simple to warrant that, I was told. They do have dset to look at
the ESM logs but not for CentOS nor Fedora. Redhat is their
"validated" [sic] OS. That's the only one they support. So I'm sort of
stuck there.
> Also check on the temperature of the machines. We
>have had some pretty wierd issues with ram and CPU quirkyness when
>they reach a high internal temperature. If you can do some poling
>using ipmi on the nodes to record the current temp and fan data over
>time so that you could see what it was at just before a crash you
>might be able to point it to an environmental situation.
I'll try ipmi. I was trying lm_sensors but apparantly it does not have
a driver for this chipset / motherboard combination. Not sure if its
an AMD Opteron specific driver issue or a
vendor-not-relesing-motherboard-specs issue (heard both versions on
the net). Anybody else had success using lm_sensors on the SC1435?
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rpnabar at gmail.com Fri Oct 10 11:05:34 2008
From: rpnabar at gmail.com (Rahul Nabar)
Date: Fri, 10 Oct 2008 10:05:34 -0500
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
Message-ID:
>(1) Tell your Dell salesman that you have asked for help on this problem on
>a public mailing list for High Performance Computing. Tell him/her that you
>need high level Dell support on this. There are Dell customers on this list.
Thanks John. I will do that. A question: how likely is it that this is
a software issue and not hardware from my symptoms? They keep harping
on the fact that I am running a non-validated OS. We used to run
Fedora. Now run CentOS. Same issues. They only support RedHat. I have
a hard time being 100% certain but the more I see it the more I am
convinced it is the hardware.
>(2) Suspect the RAM. Ask some serious questions of your Dell support about
>RAM compatibility - HPC applications stress the RAM. Ask, and ask again, if
>the specific RAM chips you have are certified for that motherboard. Use
>dmidecode to read out the manufacturer codes of the RAM modules - do you
>have a mix of manufacturers?
Very good idea. Never tried that. I will check. I assumed that they
were all similar systems and I had compatible RAM since I bought it
all packaged together.
>Ask and ask again about BIOS updates being available for these machines.
>We had a case once of HP machines - even though the BIOSes were versioned
>the same on 200 machines, there were some differences - turns out you had to
>go as far as checking the build date.
>Get the very latest BIOS version you can.
I have the latest. But that's only based on the version #. I will dig
deeper. Could this be bad BIOS, though, from the symptoms? So, some
code somewhere switches the state of that LED from blue to orange and
if only I knew what the trigger was supposed to be. Someone had to
write that!
>
>(3) The RAM will be the problem - but if you can keep notes and there are
>specific machines which crash more than others point this out to Dell and
>maybe suspect the PSUs being weak on those machines.
Yes. The crashes seem to be very clustered. We have had 5 specific
machines out of 23 crash repeatedly. We swapped the motherboard+cpus
on those and they do not seem to have crashed again as yet. But the
time scale is only about 2 weeks. So I am not very confident of the
statistical significance of my conclusions.
--
Rahul
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Fri Oct 10 11:29:00 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Fri, 10 Oct 2008 16:29:00 +0100
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <9f8092cc0810100829w3feca740j79c1822fcb7749f1@mail.gmail.com>
2008/10/10 Rahul Nabar
>
>
> Yes. The crashes seem to be very clustered. We have had 5 specific
> machines out of 23 crash repeatedly. We swapped the motherboard+cpus
> on those and they do not seem to have crashed again as yet. But the
> time scale is only about 2 weeks. So I am not very confident of the
> statistical significance of my conclusions.
>
>
>
Rahul, run that past us once more please.
You have bought 23 machines and FIVE of them have had CPUs and motherboards
replaced?
Errrrr...
By the way, I live in London, UK. The news is our local government had lots
of money in an Icelandic bank, which they've now lost. As a consequence,
they are looking to sell off objects of historic significance which they
own. I can put you in touch if you are looking for a Victorian-era lifting
bridge.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Fri Oct 10 11:22:40 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Fri, 10 Oct 2008 16:22:40 +0100
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com>
2008/10/10 Rahul Nabar
>
>
> Thanks John. I will do that. A question: how likely is it that this is
> a software issue and not hardware from my symptoms? They keep harping
> on the fact that I am running a non-validated OS.
I have had that line from a few companies.
Looks like you are talking to first or second line support people - this is
what they have been trained to say.
You can understand this - it stops idiots calling in to them who are trying
to run some spit-and-sawdust distribution they got free with a packet of
breakfast serial.
Again, just use your salesman or account manager and say that you spent $$$
with Dell to run applications,
and that the systems were sold to you as being able to run Linux.
> I have the latest. But that's only based on the version #. I will dig
> deeper. Could this be bad BIOS, though, from the symptoms? So, some
> code somewhere switches the state of that LED from blue to orange and
> if only I knew what the trigger was supposed to be. Someone had to
> write that!
The light coming on indicates a fault.
You should be interrogating the onboard management controller (called an
IPMI card ort a BMC, or in Dell speak I think a DRAC).
Log onto the suspect nodes anr run ipmitool:
ipmitool -I open sel elist
Do you not get a fault code on the little screen beside the light?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From hearnsj at googlemail.com Fri Oct 10 11:38:50 2008
From: hearnsj at googlemail.com (John Hearns)
Date: Fri, 10 Oct 2008 16:38:50 +0100
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com>
References:
<9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com>
Message-ID: <9f8092cc0810100838i4a5f24du32d9dff619ab832f@mail.gmail.com>
Actually, let's be fair to Dell here. They have replaced five machines.
Those five machines will have an up-to-date BIOS version. There may well
have been a problem which has been fixed in this BIOS version. I would use
dmidecode, and make sure that all your machines have an identical BIOS
version.
Any that do not - download and flash the BIOS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
-------------- next part --------------
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From dnlombar at ichips.intel.com Fri Oct 10 11:55:33 2008
From: dnlombar at ichips.intel.com (Lombard, David N)
Date: Fri, 10 Oct 2008 08:55:33 -0700
Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To: <2098513302.1340111223590854452.JavaMail.root@mail.vpac.org>
References: <831359607.1340021223590533957.JavaMail.root@mail.vpac.org>
<2098513302.1340111223590854452.JavaMail.root@mail.vpac.org>
Message-ID: <20081010155533.GA21681@nlxdcldnl2.cl.intel.com>
On Thu, Oct 09, 2008 at 03:20:54PM -0700, Chris Samuel wrote:
>
> What we do on our CentOS cluster is disable the standard
> screen blanker in our startup scripts with:
To clarify, see console_codes(4):
> echo -e "\033[9;0]" >/dev/console
Sets screen blank timeout to 0
> echo -e "\033[13]" >/dev/console
Unblanks the console
--
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Fri Oct 10 16:28:10 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Fri, 10 Oct 2008 13:28:10 -0700
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <20081010202810.GD11550@bx9.net>
On Fri, Oct 10, 2008 at 09:54:35AM -0500, Rahul Nabar wrote:
> I'll try ipmi. I was trying lm_sensors but apparantly it does not have
> a driver for this chipset / motherboard combination.
The CentOS lmsensors package is pretty old. You can get a more
up-to-date one (2.10) from
http://atrpms.net/dist/el5/lm_sensors/
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Fri Oct 10 16:36:21 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Fri, 10 Oct 2008 13:36:21 -0700
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <20081010203621.GE11550@bx9.net>
On Fri, Oct 10, 2008 at 09:47:32AM -0500, Rahul Nabar wrote:
> Thanks Greg. I suspected that....I am actually curious: when exactly
> can kdump be useful?
For software bugs, but probably only if you're a kernel developer.
Otherwise just capturing the oops or printks is most of what you'd
want to know.
> >Probably your fastest solution is to swap parts until works. Tedious,
> >but...
>
> That's exactly what I'm doing so far! :-) Problem is which ones?
All of them, of course. If you can cause the crash fast enough, it's
easy. It sounds like your crash happens pretty slowly, which is the
hardest kind of problem to diagnose.
-- greg
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From andrea at soe.ucsc.edu Fri Oct 10 03:52:33 2008
From: andrea at soe.ucsc.edu (Andrea Di Blas)
Date: Fri, 10 Oct 2008 00:52:33 -0700 (PDT)
Subject: [Beowulf] large MPI adopters
In-Reply-To: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl>
References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu>
<0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com>
<3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl>
<48EB69B6.5080602@tamu.edu>
<57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl>
Message-ID:
hi dan, vincent, gerry, bodgan, and all of you,
first of all, thank you very much for your answers - I really appreciate.
what I understand from your messages is that there isn't what I was
looking for - or maybe that my question wasn't clear enough.
what I was wondering about is if there are any MPI-based applications that
are well known, either because they are being sold as such, or because
they are known to be used by (large) companies. for instance, we all know
that google uses mapreduce on top of gfs. we know that yahoo uses a
version of mapreduce called hadoop. anything similar for MPI?
or - are there any commercial products based on MPI that one could buy and
run in their own company/institution? for example, let's say I want do do
some simulations of whatever kind, and I like matlab. can I go buy a
cluster of some kind, and also buy some matlab implementation that will
use MPI to run on my cluster? or some other MPI-based tool? is all
software like that still for internal use only of the entities that
develop it?
thanks again,
andrea
--
Andrea Di Blas UCSC
School of Engineering
Tel: (831) 459 4193 Fax: (831) 459 4829
On Wed, 8 Oct 2008, Vincent Diepeveen wrote:
>
> On Oct 8, 2008, at 4:09 PM, Bogdan Costescu wrote:
>
>> On Tue, 7 Oct 2008, Vincent Diepeveen wrote:
>>
>>> Drug type companies eat typical 0.5% system time.
>>
>> Could you please explain what you mean here ? That they buy lots of CPUs
>> but only use a fraction of them or only for a fraction of time ?
>>
>
> When you add up others.
>
> Note i see in IWR you got a 160 node opteron cluster, what software eats
> system time on that cluster?
> There must be graphs about it, want to share one? Also interesting to know is
> how many cores most jobs eat!
>
> Note that i left out 1 important factor which is not a factor in every
> nation: that's the amount of system time airindustry eats of supercomputers
> designing new stuff. It is very significant together with car manufacturers.
> With 160 nodes machine, even if that would be power6 nodes,
> you can't even design a wing for aircraft.
>
>>> They are far behind, which is not so logical; i would consider it old
>>> fashioned to do experiments on monkeys (Bar-ilan brain experiments on
>>> monkeys rings a bell?), so doing everything in software is far more
>>> logical.
>>
>> Maybe your knowledge about the pharma industry is far behind ? "Doing
>> everything in software is far more logical" only when you can exactly
>> express biological processes in mathematical terms which is not true in the
>> great majority of cases. F.e. molecular dynamics (often used to simulate
>> interactions at molecular level, f.e. of a drug injected into the body) is
>> based on several approximations and the implementations are subject to
>> numerical errors. Would you take a drug that was produced based only on a
>> simulation with molecular dynamics and found to be good enough ?
>>
>
> Well, i sure avoid hospital visits, you?
>
> As you know the healthcare commission is basically old men who spread careful
> chosen sentences in such a manner that
> everything looks fine on this planet and that there is nothing to worry
> about; last time i spoke with them on EMF's a guy being spokesman
> on this subject, he didn't even know how the electricity system in his own
> house worked (whether on the first floor of his house
> he had earth).
>
> So lucky to introduce to a new drug, there is some burocratic systems. There
> is several phases of testing that a drug
> has to pass in order to get accepted as a drug that is legal. Not to confuse
> with the 'advice' on whether a drug gets prescribed;
> they want way cheaper drugs for that usually than the ones that might maybe
> be better (say 0.5% better for a lot of money extra,
> but if you're that guy that didn't get cured as a result of that, your family
> will regret it bigtime).
>
> Computing power or not, most manufacturers simply want to sell a product, so
> the amount of systemtime invested on a computer,
> in order to sell, is total irrelevant to them to get a drug accepted usually.
> The first and most important part of the show to get a drug
> accepted is getting invited to the table to show your drug. Computing power
> is nice to mention here when that is usually not a factor
> at all to get a new medicine accepted.
>
> But it may change, let's hope so...
>
> Maybe a new breed of company can do the job. A breed where making profit is
> not most important :)
>
>> Similar concerns apply to all levels - the brain, as you mention it, is
>> still very much an enigma, otherwise artificial intelligence would be all
>> around us (or not, if deemed to dangerous...). And when you don't know how
>> an organ or system works, how can you develops drugs only by simulations ?
>>
>>> When someone is really in big need for big calculation power, one makes
>>> dedicated cpu's for it.
>>
>> I tend to agree, provided that the dedicated CPU brings a really big
>> advantage, like an order of more of magnitude. F.e., to stay in the pharma
>> related field, that's the bussiness case for DEShaw Research's molecular
>> dynamics chip (Anton) which is supposed to bring a significant increase in
>> speed for MD simulations compared with software solutions running on
>> general purpose CPUs (or GPUs).
>
> Some hardware guys can better comment here. Making your own FPGA chippie is
> rather cheap.
> Finding and paying a few good guys to get the chip done is toughest. You
> basically need 1 good
> programmer, the rest can get hired parttime in fact. After chip works,
> to print a 1000 cards, it is like a 100 dollar for pci-card and 50 dollar a
> chip.
>
> So $150k for computing power that total annihilates any supercomputer of now
> and the NEAR future for your specific application.
>
> Real cpu's are a lot faster of course, price for them i do not have at hand.
> I heard it could be cheaper
> than this in fact if you regurarly print a batch. Such a batch is producing
> less cpu's than the FPGA, where you pay for 1 cpu a constant
> amount. I tend to remember that good cpu designers, not to mention memory
> controller guys, are rather expensive folks, as
> they are very popular; on some other mailing list with some very funny
> lastnames, i see nonstop job offers for that type of folks.
>
> Vincent
>
>
>> --
>> Bogdan Costescu
>>
>> IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
>> Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
>> E-mail: bogdan.costescu at iwr.uni-heidelberg.de
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From rlinesseagate at gmail.com Fri Oct 10 11:19:14 2008
From: rlinesseagate at gmail.com (Rob Lines)
Date: Fri, 10 Oct 2008 11:19:14 -0400
Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes.
In-Reply-To:
References:
Message-ID: <69592cde0810100819w7dbae26bi6e162445cb3ada88@mail.gmail.com>
On Fri, Oct 10, 2008 at 10:54 AM, Rahul Nabar wrote:
>>Have you checked in the baseboard management log to see if it is
>>throwing an error.
>
> Apparently the SC1435 does not have OpenManage. "Simple Computing" is
> too simple to warrant that, I was told. They do have dset to look at
> the ESM logs but not for CentOS nor Fedora. Redhat is their
> "validated" [sic] OS. That's the only one they support. So I'm sort of
> stuck there.
We have a cluster of SC1435 without the DRAC card but they still have
the baseboard management. It should be accessible right at the end of
the POST by hitting ctrl+e and normally it shows a little splash
screen with the ip it has been configured with and what keys to press.
It gives some great information especially if you are running into
ram issues. I don't remember if we had to run on the logging or if it
was on from the factory.
>
> I'll try ipmi. I was trying lm_sensors but apparantly it does not have
> a driver for this chipset / motherboard combination. Not sure if its
> an AMD Opteron specific driver issue or a
> vendor-not-relesing-motherboard-specs issue (heard both versions on
> the net). Anybody else had success using lm_sensors on the SC1435?
We never tried to deal with lm_sensors on these machines because ipmi
had more options. We are using CentOS 5 on our cluster machines.
Here are a few links we used when working on ipmi.
OpenIPMI
http://www.barryodonovan.com/index.php/2007/04/11/dell-ipmi/http://66.102.9.104/search?q=cache:IoPmIimMExQJ:www.barryodonovan.com/index.php/2007/04/11/dell-ipmi/+linux+ipmi+temperature+sensor+read&hl=en&gl=us&strip=1
http://www.hollenback.net/index.php/LinuxServerManagementIpmi
http://openipmi.sourceforge.net/
http://ipmitool.sourceforge.net/
http://linux.dell.com/files/presentations/Red_Hat_Summit_May_2006/ipmi_presentation-redhat_summit.pdf
http://buttersideup.com/docs/howto/IPMI_on_Debian.html
>We used to run Fedora. Now run CentOS. Same issues. They only support RedHat. I have a hard time being 100% certain but the more I see it the more I am convinced it is the hardware.
The real key is to go back to the sales person that you purchased them
from and tell them the problems you are having and have them help
escalate the situation. I am not sure what diagnostics you are
running but they have a bootable cd for doing a memory test at the
link below. Even when they have tried to point at an OS problem if I
can get an error on that cd they really have nothing that they can
hide behind.
http://support.dell.com/support/downloads/format.aspx?c=us&cs=19&l=en&s=dhs&deviceid=196&libid=13&releaseid=R169189&vercnt=5&formatcnt=0&SystemID=PWE_XEO_1435SC&servicetag=85PG2F1&os=LIN4&osl=en&catid=-1&impid=-1
Best of luck,
Rob
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
From lindahl at pbm.com Fri Oct 10 16:53:00 2008
From: lindahl at pbm.com (Greg Lindahl)
Date: Fri, 10 Oct 2008 13:53:00 -0700
Subject: [Beowulf] large MPI adopters
In-Reply-To: