ScaLAPACK Intro: Part I of IV

ScaLAPACK (
http://www.netlib.org/scalapack/
), the "Scalable Linear Algebra Package," is the distributed-memory follow-on to LAPACK. Although source code for ScaLAPACK and the lower-level libraries on which it depends, like BLACS, BLAS, and PBLAS are freely available via netlib, the major HPC vendors put considerable effort into optimizing these routines for their specific architectures. (Cray includes ScaLAPACK routines in craylib. IBM offers the ScaLAPACK functionality through a different API and set of libraries, PESSL.)

If your code calls LAPACK routines, like SGETRF and SGETRS, and you want to dramatically expand your problem size, you might consider parallelizing it to use the corresponding ScaLAPACK routines, like PSGETRF and PSGETRS.

This series of articles demonstrates the conversion of one very simple LAPACK code to ScaLAPACK. You'll immediately discover that the greatest hurdle to using ScaLAPACK is not in calling the routines, but in set-up.

The parallel solvers expect that:

the processors are arranged in a "virtual processor grid," and

the data are physically distributed amongst the processors in block-cyclic layout.

The concept of a "virtual processor grid" is already known to all MPI users. When you launch an MPI program with NPROCS tasks, they are arranged for you in a 1D grid, counting from 0 to NPROCS-1. E.g., if NPROCS were 6, the tasks would be arranged as: 0 1 2 3 4 5
Each task knows its unique number in this 1D grid and the extend of the grid (i.e., NPROCS).

ScaLAPACK is built on BLACS (
http://www.netlib.org/blacs/index.html
), the "Basic Linear Algebra Communication Subprograms," which is similar, but arranges processors in a 2D grid. E.g., if NPROCS were 6, the tasks might be arranged in a 2x3 grid, counted as:
(0,0) (0,1) (0,2)
(1,0) (1,1) (1,2)
Each task knows its unique row and column number in this 2D grid and the extent of the grid (i.e., NROWS and NCOLS).

The concept of block-cyclic data distribution is far more difficult to visualize. We'll touch on it, but the example in this series of articles uses utilities from the "ScaLAPACK Tools" library to hide most issues of the block-cyclic distribution from the programmer.

What follows is our example LAPACK program. It initializes and solves AX=C, where A is an n x n matrix, and X and C are vectors of length n.

In Part II of this series, we'll start converting this code to run on a distributed memory parallel processor, using ScaLAPACK.

IBM Lightweight Core Files

Core files can be a real time saver when try to track down bugs in your code. But traditional core files require a debugger to get useful information out of them. There is another alternative for MPI programs on IBM systems such as iceberg and iceflyer- lightweight core files. These core files are significantly smaller than traditional core files and provide a stack trace and other useful information in human readable form.

Here's how to dump lightweight core files:

Compile with an mpi compiler (e.g. mpxlf_r, mpcc_r, mpCC_r, etc.)

Add -g to the compiler flags so that debugging information will be added to the executable. Without this flag line numbers and other useful information will not be included in the stack trace.

E.g.:
iceberg1 37% mpCC_r example.cpp -g -o example

Add the environment variable MP_COREFILE_FORMAT=STDERR to your loadleveler script. This will cause the lightweight core information to be written to standard error. You may also want to add MP_LABELIO=yes in case more than one MPI task dumps core (MP_LABELIO=yes prepends the task number writing output to the each line of output). E.g.:

If the application crashes, it will now write the lightweight core file to standard error. Below is sample output from the example application. If you have set MP_LABELIO to yes, you can now look at the output for individual tasks using grep.

Note that you can get lightweight core files, even if you don't compile with -g. You'll get less information, but better performance. We recommend that you do not compile with -g for production runs!

Checking Your Disk Quota on the ARSC IBMs: qcheck

ARSC provides a locally written script, "qcheck," for checking your disk quota on the iceberg and iceflyer. This makes it as easy to check your quota on the IBMs as on the Cray, where you should use "quota -v".

Avoid programming languages that lend themselves to contests of guessing what a line of code means.
-- Corollary to Murphy's Law.

Perl is an abominable language. It is, unfortunately, the lingua franca of Unix scripting. Because we need it all the time and because of its usefulness in string processing, it is a necessary tool for our kit.

There are scores of books on Perl -- a search of Amazon.com returned over 7800 hits on "perl." This book is for people who know Perl and don't want to wade through hundreds of man pages to answer questions.

Perl in a Nutshell does a good job of covering most aspects of the language, from its basics to its modules and connections to CGI. Whenever I am forced to use Perl, it is the book I have on my desk. It is useful for checking items from the correct syntax for concatenation, where no two languages seem to agree, to the proper routines and modules for file operations or large number arithmetic.

A nice feature is the chapter, "Function Reference." Perl functions are listed by category and then alphabetically. When I need to open a file, for example, I go to "File Functions" at the beginning of the chapter to see my options. When I want the exact syntax and semantics of the "substr" function, I go straight to it in the alphabetical listing.

A few chapters later, they have a similar structure for the hundreds of standard modules. When I needed to handle many-digit numbers, for example, I found "bigint" and "bignum" modules there, which provided unlimited precision integers. Overkill for my use, but it worked. The reference was crucial at that time and easy to stumble across in the book.

One weak aspect of the book is the index. It does not list as many variations on terms as I need. I've added my own cross references to normal terms for various items, diacriticals and punctuation marks. For example,

The list of symbols in the index omits the three (very) different types of quote marks.

There are all sorts of references to the Math modules, but none to the common max and min functions.

There is an index entry for the "sleep" function, but no reference from such common synonyms as "pause" or "hibernate." The more Perl programming a person does, the less important the missing synonyms in the index.

Santa Letters, Postmarked "North Pole"

Uncles, Aunts, Parents, and other friends of kids: here's an offer from the ARSC HPC Newsletter editors.

The town of North Pole, Alaska is a mere 15 miles from here. If you'd like a letter, postmarked "North Pole," delivered to someone, seal your letter in a stamped, addressed envelope, and instead of mailing it from your local post office, enclose it in a larger envelope and send this to us. On about December 13th we'll mail the letters from North Pole.

Plan extra time for mail to/from Alaska... If you post these to us before Dec. 6th, there should be no problem.

Quick-Tip Q & A

A:[[ I'm looking for an easy, automatic way of scanning through a list of
[[ files to change all lines like this:
[[ #include "some_string"
[[ to this:
[[ #include <some_string>
#
# Richard Griswold
#
This is easy to do with perl:
perl -pi -e 's/(#include )"(some_string)"/$1<$2>/' file ...
Since you are using perl's s/// operator, you have a full range of regex
options available to you. If you are a bit unsure of your changes and
want to keep backups, use "-pi.bak" instead of "-pi" on the command line.
You can also combine this with find and xargs to change files in multiple
directories:
find . -name "*.cpp"
xargs perl -pi -e 's/(#include )"(some_string)"/$1<$2>/'
#
# Kate Hedstrom
#
A Perl one-liner:
perl -i -pe 's/#include "([^"]*)"/#include <$1>/' file.c
The pattern matches double quotes around any number of non-quote
characters and stores it in the $1 buffer. The -i option means to
replace the file with the changes. Otherwise it would be:
perl -pe 's/#include "([^"]*)"/#include <$1>/' file.c > file_new.c
#
# Dale Clark
#
Here's one solution:
perl -pi -e 's/#include "some_string"/#include <some_string>/g' \
`find . -type f`
This 'find' expression would act on all plain files in the current
directory tree. Some other expression (like "ls *.c") could of course
be used in its place, as appropriate.
#
# Greg Newby
#
There are lots of standard Unix/Linux utilities to do this sort of
thing, as well as more fully-featured languages with good pattern
substitution functionality such as Perl (I expect someone else will
present a single-line Perl script to do the same thing). A simple
case would be to take each file and transform it to a new file. If
the new file is different than the old file, we presume that
substitutions took place. Replace it (perhaps saving a copy).
I'd use "sed" for this, which should be available on any Unix/Linux
system. There are some limitations to sed on some systems, but
assuming you're just talking about source code files, sed is a safe
choice (don't use sed for binary data, or files with very long lines,
without testing first).
Here's a quick little script for all .cpp files:
#!/bin/sh
# Loop for each .cpp file:
for i in *.cpp ; do
# Send the file as input to "sed". We need a backslash
# before special characters such as ", #, < and >:
cat $i
sed 's/\#include \"some_string\"/\#include \<some_string\>/' > /tmp/$i
# See whether the newly transformed files are different than the old:
diff $i /tmp/$i > /dev/null
if [ $? -ne 0 ] ; then
cp $i ${i}.old # Make a backup
cp /tmp/$i $i # Copy the changed file over the old file
fi
done
#
# Wendy Palm
#
It sounds like sed with some regular expressions may be one way to go.
This miniscript will change any line that has #include "...." to
#include <....>, accounting for an arbitrary number of whitespace
characters (anywhere but leading whitespace).
#! /bin/sh
for file in $*
do
cat $file
sed '/^#[ \t]*include[ \t]*/s/\"/\</'
sed '/^#[ \t]*include[ \t]*/s/\"/\>/' > $file$$
mv $file$$ $file
done
Q: I was recently on a walk and discovered a phone number, that I
wanted to remember, posted on a tree. I had time to cogitate, but
nothing on which to write.
Do you have any tips, methods, or systems for memorizing numbers?
If you want, you could demonstrate your system using one or more of
the following numbers (these are the ARSC Consult line, our latitude,
and the ISBN for everyone's favorite book, "Parallel Programming in
OpenMP").
907-450-8602
64 deg. 51.590 min.
1-56592-054-6

The University of Alaska Fairbanks is an affirmative action/equal
opportunity employer and educational institution and is a part of the University
of Alaska system.
Arctic Region Supercomputing Center (ARSC) |PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8602 | fax: 907-450-8601 | Supporting high performance computational research in science and engineering with emphasis on high latitudes and the arctic.
For questions or comments regarding this website, contact info@arsc.edu