Contents

tcsh "autologout," Especially on the ARSC IBMs

UNIX shells always seem to find a way to keep you on your toes. If you use tcsh and regularly, but mysteriously, find yourself logged out of terminal sessions, the problem might be the autologout functionality. When the variable "autologout" is set, the tcsh shell will *automatically* disconnect the session after the specified number of minutes. E.g.:

# first set the autologout variable to 60 minutes.
iceberg1 3% set autologout=60
# verify that it is set using the built in shell command, 'set'.
iceberg1 4% set
grep auto
autologout 60

In this case the shell will disconnect after 60 minutes of idle time. After the specified amount of idle time, we would see the following:

iceberg1 4% auto-logout
Connection to iceberg closed.

In some versions of tcsh including those installed on iceberg and iceflyer, the autologout variable may be set by default (!!!) to 60 minutes. Sifting through the tcsh man page, one finds that autologout is set to 60 minutes by default in login and superuser shells, but not if the shell "thinks" it is running under a window system. It checks the setting of the environment variable DISPLAY to assess whether or not the session is running under a window system.

Thus, the autologout variable could be set for you automatically:

If you use a terminal program which does not support X11.

If you use a terminal program which does support X11, but you are using a connection method which does not set the DISPLAY environment variable for you (e.g., krlogin, ktelnet).

Note: ssh sets DISPLAY, as long as the terminal supports X11.

Fortunately, if, for some crazy reason, you don't want to be logged out automatically, you can get around this by unsetting the autologout variable. E.g.:

unset autologout

This command can be added to your .cshrc if you would like to permanently disable autologout.

Remember this only applies to the tcsh shell.

ScaLAPACK Intro: Part III of V

In this issue, we finally begin distributing the data arrays across all processors. This is where the dream of performing linear algebra on large arrays, say, 300000 x 300000 64-bit REALs, starts to become reality. For the record, a couple of arrays this size would indeed fit on ARSC's IBM cluster, with room to spare.

As described in
Part I
, ScaLAPACK requires that the data arrays be block-cyclically distributed across the 2D processor grid (which was created in
Part II
). For each array, we need to take the the basic steps:

settle on a block size

call DESCINIT to create a standard ScaLAPACK array descriptor

allocate local memory for each process' portion of the array

distribute the actual data values into the allocated memory

(I've decided to save step 4 for the next issue--which is why this series has just expanded from IV to V parts). So, step 1:

settle on a block size

Yes, the programmer, not ScaLAPACK, determines the block size. "Block size" refers to the dimensions of the subdivisions into which the global array is decomposed. These blocks are then dealt out to processors like playing cards being dealt in a game of canasta. As an example, here's a 4x5 array:

00 01 02 03 04
10 11 12 13 14
20 21 22 23 24
30 31 32 33 34

Given a block size of 2x2, this array would decompose into the following six blocks:

00 01
10 11
02 03
12 13
04
14
20 21
30 31
22 23
32 33
24
34

The ScaLAPACK User Guide recommends a block size of 64x64 for large arrays, and in my rough experimentation on both the X1 and IBM cluster, 64x64 was indeed best.

To compute the block size in the sample code, I've borrowed another subroutine from Carlo Cavazzoni of CINECA. "Blockset" chooses a good block size based on the size of the global array, and number of row and columns in the processor grid. It also honors a maximum block size value, which, following the User Guide recommendation, can simply be set at 64.

call
DESCINIT
to create a standard ScaLAPACK array descriptor

At this point, we have this information:

dimensions of the global array

block size

BLACS context

We only need two additional bits of information to complete the description. The first is easy, the processor that will own the first element of the array, which we arbitrarily set to the processor with grid coordinates (0,0). The second is more interesting. From the "man" page for the ScaLAPACK Tools routine, DESCINIT, we need this:

lld Integer. (input) The leading dimension of the local array
that stores the local blocks of the distributed matrix.

Each process can get the "lld" of the array using another Tools routine, NUMROC ("NUM Rows Or Columns"). This is described in the IBM PESSL manual:

"This function (NUMROC) computes the local number of rows or columns of a block-cyclically distributed matrix contained in a process row or process column, respectively, indicated by the calling sequence argument iproc."

Since each process only needs the local leading dimension for DESCINIT (i.e., the local number of ROWs), forget about columns, and call NUMROC with input arguments for:

the number of ROWs in the global array

the number of ROWs in block size

the total number of ROWs in the process grid

the local processor's ROW number in the process grid

and NUMROC will magically return:

the total number of ROWs in the local portion of the distributed array. In other words, NUMROC returns the local leading dimension.

Whew! We can now call DESCINIT, which returns the required array descriptor for all subsequent ScaLAPACK calls involving the array.

allocate local memory for each process' portion of the array

In my trivial "block-size" example, above, the 2x2 block size did not divide the 4x5 data array into equally sized blocks. One outcome of this fact is that if we distributed the 6 blocks onto, for instance, 6 processors, the local storage required for this array would vary between the processors. In general, the local storage will indeed vary between processors.

Luckily, we're only missing one bit of information to determine each processor's exact local array dimensions.

Having already called NUMROC to determine the local number of rows, we simply call it again (with basic column information as input) to obtain the local number of columns.

Given the local dimensions, each process uses Fortran ALLOCATE to allocate its portion of the array in local heap memory. The sum of all the memory used in the local arrays will add up to the size of the global array.

---

The test code, below, now does everything through array allocation. I've added a subroutine "printlocals" to display information about the distributed arrays. Here's program output, with some explanations interspersed:

The next block of output, above, shows that the global dimensions of array A are (13 rows : 13 columns). Each process also tells us that the block size is 4 and the total number of rows and columns in its local portion of the array. For instance, proc (0,0) has 5 rows and 8 columns of "A."

In the block-cyclic distribution, this first row of blocks is dealt out entirely to the first row of processors in the processor grid. Thus, given our 3x2 grid of processors, these four blocks would be assigned as follows:

The number of columns is the last value on these two lines, and, yes, processor (0,0) indeed reports 8 columns and processor (0,1) reports 5. We could perform a similar analysis on the first column of blocks to see how it's dealt out the the first column of processors. We could do the same thing for the entire array.

If this explanation leaves you confused, be sure to check out the ScaLAPACK User Guide, the CINECA tutorials, or other resources. Here's the updated code:

In Part IV of this series, we'll actually populate the allocated local arrays with data. Can you feel the excitement starting to build?

NCL Upgrade Announced

The NCL developers have issued the following announcement. This isn't installed at ARSC yet. NCL users should let us know if they're interested.

> We are pleased to announce that a new version of NCL (4.2.0.a032)
> is now available for download:
>
> http://ngwww.ucar.edu/ncl/download.html
>
> This version has the long-awaited feature of being able to generate
> contours on non-uniform grids. This is probably the most significant
> functionality added to NCL since its release in 1995.
>
> For more information on the types of grids that NCL can now contour
> and to view some examples, see:
>
> http://ngwww.ucar.edu/ncl/grids/
>
> Since this new functionality has only been tested on a handful of
> grids, we are asking the user community to help us test this new
> capability. We will be happy to help you get started.
>
> This version also has several new processing and graphics functions
> and capability enhancements. To list a few:
>
> - a much faster version of the EOF routines
> - functions for writing Vis5D files
> - an enhanced GRIB reader
> - function for assigning randomly spaced data onto the nearest
> locations of a grid with two-dimensional latitude and longitude
> arrays
>
> For the full list of what's new, see:
> http://ngwww.ucar.edu/ncl/whatsnew.html

OpenMP 2.5 Draft Specification Released for Public Comments

> The OpenMP ARB is pleased to announce the release of a draft of the 2.5
> specifications for public comment. The goal of the 2.5 effort was to
> combine the Fortran and C/C++ specifications into a single one
> and to fix inconsistencies. The language committee has done a phenomenal
> job for the last 18 months in combining the old specifications -- every
> word has been carefully reviewed and almost the entire specification has
> been rewritten.
>
> The ARB warmly welcomes any comments, corrections and suggestions you
> have for Version 2.5. Please send email to feedback@openmp.org. It is
> most helpful if you can refer to the page number and line number where
> appropriate.
>
> The public comment period will close on 31 January 2005.

Holiday Greetings

Happy holidays, everyone. Here's a seasonal image for you from the land of winter... (this image was submitted with the photography "Quick-Tip", below)

Quick-Tip Q & A

A:[[ Holiday fun... Share a tip on ANY SUBJECT! Anything! Teach us
[[ something. What do you do well?
#
# Thanks to all for the big response!
#

Talk less listen more.

Frequent second hand bookshops.

Recycle if you have to, but it would be better not to use in the first place.

Here is a quick tip on photography. For best Christmas lights photos, shoot them during the twilight hours (between 2-3pm in Fairbanks, this time of year).

Review your charitable giving, of both time and money. Are you giving enough? What's important to you? This is the perfect time to reconsider reasons and goals and make a plan for the new year.

Best mittens to knit - Lincoln wool on the outside for toughness, alpaca/merino yarn on the inside. Pattern is from the Norwegian issue of Knitter's magazine: two-end knitting in worsted weight, tall (3-4 rows) duplicate stitch lining, then turn inside out.

If you are a spinner with a drum carder, try blending different fibers. Wool with dog hair, mohair, silk, or alpaca, all are really nice. The blend will have some of the best features of each fiber - soft wool gives elasticity to almost everything.

When baking cakes and cookies, people tell you to measure carefully. What is important to the texture is the moisture/flour ratio. The amount of sugar is less important and can be vastly reduced without changing anything but the sweetness.

If you are out in the cold and you are having trouble touching your pinky to your thumb (on the same hand), then it is necessary to stop and start a fire immediately. If you wait much longer, you may not have enough dexterity to use a lighter or to light a match.

Should you become lost in the wilderness, be sure to eat more than just rabbits (or hares). There is so little fat in these critters that you will actually die within a week or two. This is known as "rabbit starvation" and will start out as diarrhea, followed by a great hunger that cannot be satisfied. This leads to the eating of more and more rabbits in a vain attempt to stop the hunger. Death soon follows.

Wear gloves while cleaning rabbits, avoid rabbits with spots on their livers, and don't eat visibly sluggish rabbits. This will help to prevent the infectious disease tularemia, which many rabbits carry.

On those long sub-zero walks with your dog, put vaseline on the metal collar ring and leash snap, and Fido's tongue won't stick and freeze to the metal.

I have found that when one of my dogs eats something he shouldn't have, necessitating making him take some hydrogen peroxide to induce vomiting, it's absolutely no problem at all to get him to take the hydrogen peroxide if I mix it with some yogurt. The old method of forcing the stuff down his throat just didn't work very well, and this new way seems like a treat to my dog.

Here is a way to get rid of candle wax drips on fabrics. Place a brown paper bag over the cooled wax drippings, iron with a hot iron. The bag absorbs the wax and allows you to lift it away!

WD-40 is probably the best solvent for nasty stuff like tree sap and bubble gum. Just spray it on and rub like mad, then wash the WD-40 off with water and detergent and the crud will come out like magic.

Fleece dog booties make good mittens and "slippers" for babies. The velcro holds them on and they come in packs of four.

How to get warm: Sometimes you can bundle up with lots of clothing, but still get cold.

My tip is to get a move on! If you move around (run, jump, bend over, dance, etc.), you'll find that all those warm clothes do wonders at keeping the heat in. I find that only a minute or two's worth of exertion can get me warmed up and toasty for quite some time.

At the first hint of a sore throat: Reduce your alcohol, coffee, and sugar intake as much as possible, take high concentration zinc lozenges, get rest, and drink lots of water. The most important thing, if applicable: eliminate ALL salty snacks from your diet.

If your car gets stuck in the snow, you may be able to use your floor mats to get traction. Flip them over, and stick them in front of your tires.

Make a list of all the movies you want to see on www.imdb.com and consider joining a mail order movie house like www.netflix.com. Don't waste time in the store; browse with intelligence on the net.

How many countries are there in the world? Check out the UN; add in a few that are missing. The result? Less than 200! Make a plan to visit at least one next year.

To revive hard brown sugar, place it on a baking sheet and sprinkle with water. Set in the oven on the lowest setting for about 10 minutes. This should make it soft enough to mash it with a spoon or a rolling pin. If there are still hard pieces, repeat.

To get an avocado out of the skin easily, cut it in half and then use a spoon to scoop between the skin and the flesh.

To make pots-and-pans cleanup easier when cooking over a campfire, coat the (outside of!) the pan with dish soap. The smoke stains will wash off much easier.

Peel bananas as apes do. Pinch the end of the banana opposite the stem and it opens easily every time.

The University of Alaska Fairbanks is an affirmative action/equal
opportunity employer and educational institution and is a part of the University
of Alaska system.
Arctic Region Supercomputing Center (ARSC) |PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8602 | fax: 907-450-8601 | Supporting high performance computational research in science and engineering with emphasis on high latitudes and the arctic.
For questions or comments regarding this website, contact info@arsc.edu