Jobs

Introduction

Anyway, there’s a reason to return to these pages and write another installment. And the reason is in the title. After some work that we’re going to get to in a minute, I’ve managed to get to the point where I can submit and successfully execute jobs on the COS operating system. As things stand today, this is most likely also the last piece to this series, but more on that later.

The filesystem

Before we get into how batch processing works, lets talk a little bit about how files and the file-system works on COS!

Modern filesystems go to great lengths to organize your files into logical groups. They provide directories, symbolic and hard links, and long file names. At the same time, files themselves are just binary blobs and the interpretation of them is completely delegated to the applications using them.

COS calls files datasets, and (as far as I can tell) gives you very little in terms of organizing them. Permanent dataset names can be up to 15 character long and there are no directories. You could add a rather long note to describe what a dataset was about though. At the same time COS had a rich understanding of the internals of datasets. They are broken into files, and further into records. Records can have variable lengths so can have files.

Datasets are maintained in ‘Catalogs’, which are somewhere in between directories and file-systems in modern terms. The Dataset Catalog (DSC in short) contains an entry for every permanent dataset that is stored on the hard drive(s). There are two optional but highly recommended catalogs, the Master and Backup catalog.

The backup catalog contains entries for datasets stored on backup tapes. In addition to the usual data, like the name of the dataset it also contains an edition number, essentially implementing versioning of backups.

The master catalog serves as a set of links to gather together all editions of a dataset into a single location. It might point to the DSC entry for permanent datasets or to the backup catalog for archived datasets. This feature deeply integrates backup handling into the OS. For example if a job references a dataset that’s not available on permanent storage, an operator is notified to load the right backup tape into one of the tape-drives to make it available for the job to access.

There’s a third interesting feature of the way COS manages datasets. When a job wants to access a dataset, first it must be made ‘local’. When the job is done with the dataset, it has to explicitly release it. This – if I read things correctly – essentially implements atomic file operations, that is either all writes succeed to a permanent dataset or none do.

Batch processing

As you probably already know, the COS was a batch operating system. It doesn’t mean that it couldn’t do multi-processing, but it does mean that – at least in the way you would normally interact with it – you didn’t get a command line of any sorts. You couldn’t work on the machine interactively. What you did instead was to sit in front of a workstation hooked up to another mainframe, like a CDC Cyber or maybe an early VAX/VMS system. You run something, called the Station software. This was your interactive working environment, where you assembled your jobs into a batch, and submitted them for processing to the Cray mainframe. The mainframe then scheduled and eventually executed your job, produced the results and sent them back to your station for you to look at.

These batches of operations were described in a special language, called the Job Control Language, or JCL. To give you an idea how such a description might have looked like, here’s a job file:

1

2

3

4

5

6

<span>JOB,JN=JSYSDIR,US=CRAY,T.

ACCOUNT,AC=CRAY,APW=XYZZY,UPW=XYZZY,NUPW=XYZZY.

*

ACCESS,DN=JSYSDIR,ID=V117REL.

SUBMIT,DN=JSYSDIR.

DISPOSE,DN=$OUT,DC=SC,DEFER.</span>

Now, this might look foreign and strange, but in essence it’s not that much different from a .BAT file or a shell script. The syntax is funny, but the idea is the same: you invoke commands, some built-in, some external and pass parameters to them. There are a few specialties worth mentioning:

Since we’re operating in batch mode, what you would call today STDIN and STDOUT don’t directly exist. $IN, the equivalent of STDIN gets mapped to the job file itself and $OUT (STDOUT) gets redirected to a temporary file. When the job terminates, $OUT (along with $LOG where the system collects all sorts of statistics about what your job did) gets shipped over to your station, where it most likely gets printed.

Job files identified themselves and the user account they run under. These are the first two lines of the job file above.

Since you’re operating in an environment where your station software is running on a different machine than your actual code, you have a bit of a schizophrenia to deal with: datasets can live in two places and depending on what piece of code we’re talking about they see one set or the other. Special operations are available to transfer files between the two systems (one of these is the DISPOSE command above).

Now, I’ve mentioned earlier that datasets can contain multiple files. This feature becomes handy with job processing: you can concatenate the input to the various commands into a single dataset – one that now contains multiple files. When a new command is executed, COS advances $IN to the next file in the job dataset. Here is one example:

1

2

3

4

5

<span>JOB,JN=JGENCAT,US=SYSTEM,T.

ACCOUNT,AC=CRAY,APW=XYZZY,UPW=QUASAR.

*

FETCH,DN=GENCAT,MF=AP,AC=ST,TEXT=BIN/GENCAT.

GENCAT.</span>

This job invokes the GENCAT utility, which in turn reads its commands from $IN. So the dataset contains a second file with the following content:

1

2

3

4

<span>MCD,PDN=$MASTER,R=QUASAR,W=QUASAR,M=QUASAR.

BCD,PDN=$BACKUP,R=QUASAR,W=QUASAR,M=QUASAR.

CREATE,CAT=BACKUP.

CREATE,CAT=MASTER.</span>

These four lines inform GENCAT to generate a backup and master dataset catalogs discussed above.

Jobs can be put into job classes so they can be summarily controlled, and the number of concurrent jobs could be set as well.

The processing of the JCL file that you submit is the job of the ‘Control Statement Processor’ or CSP module. Whenever a new job is created a section of memory is carved out and set up for it by the OS (the STP component to be more specific). Since the Crays had only very rudimentary memory protection features (a per process base address offset and address space length could be set), each process needs to operate in a completely separate address space. Shared libraries are not supported. Because of this, the command processor, needs to be copied into this memory space that is reserved for the job. When additional programs get invoked (like the Fortran compiler) they also get loaded into the jobs address space.

A sample session

We start with a fully booted system (either through install or deadstart). Once, you have the system fully booted, you should see something like this on the console:

you type in the following commands:

CLASS,ALL,ON

LIMIT,5

These enable all job classes and sets the number of parallel jobs to 5.

Now, to see some status, we can enter the command ‘STATUS’:

At this point you can submit a job. There are a few on the expander disk, most of them are rather useless at least at this stage. One of those is JTEST30. Useless, maybe, but demonstrates how things work and doesn’t cause any harm. Se let’s submit it:

SUBMIT,JTEST30

At this point you’ll see the job queue – which is displayed on the screen – list your just submitted job as it paces through the various stages of execution.

Once it’s done, it will disappear from the list. So where are the results? I’ve told you that normally $OUT and $LOG goes to the printer on the station you’re using. As it turns out there was a local printer attached to the IOPs expander on this configuration. This is the location where the information was sent. The simulator dumps the output intended to the printer into PR0.TXT. So, let’s take a look at what it contains!

You can see no output from the job and the log is rather concise as well. It informs you about an error, which is that TEST30 is missing (PD000 and PD009 messages). Why is that? Let’s take a look at the job source:

1

2

3

4

5

6

<span>JOB,JN=JTEST30,T=8.

ACCOUNT,AC=ENGINEER,US=ENGINEER,APW=WITHOUT,UPW=APADDLE,NUPW=APADDLE.

*

ACCESS,DN=TEST30,ID=JON,OWN=ENGINEER.

SUBMIT,DN=TEST30.

DISPOSE,DN=$OUT,DC=SC,DEFER.</span>

Line 5, the submit statement says: load dataset TEST30 from mass storage and execute it. In our system, mass storage is empty, there’s no TEST30 to launch. Most of the jobs on the disk would behave similarly. There’s one that’s useful: JGENCAT. It generates the afore-mentioned master and backup catalogs. You can launch that too and see it eventually succeed. I’ve created a short video about job submission and execution as well, in which I do show how JGENCAT works:

Interactive mode

Did I say COS was a batch operating system and you didn’t get any sort of command prompt? Well, I’ve lied. In fact there was a way to work with the machine interactively. This mode is very similar to what you would see on a modern Unix-like machine. CSP, the command processor runs on the mainframe, but $IN, the input dataset is redirected to your console. You enter your job line-by-line, which gets immediately interpreted and executed by CSP. $OUT is also redirected to your console, so you immediately see any output. Most programs were not designed to work in this interactive environment, though there are notable exceptions: TEDI, the interactive editor, and SID, the symbolic debugger. One of these, TEDI happens to be on the expander disk, so we can see how that worked.

Getting into interactive mode is a bit more involved. First, you have to start something called the ‘interactive concentrator’ on the MIOP console. You do that, by typing in IAIOP LOG into the kernel prompt:

The concentrator – or as sometimes called the front-end interface – was a piece of hardware and software that allowed for remote stations to connect to the Crays. A non-interactive version of this was automatically started, that’s why we have been able to use the STATION command to start a station. Now, we have the interactive version running as well.

You will have type in all the usual incantations to get the OS booted and functional, most importantly, enable classes and set the job limit:

CLASS,ALL,ON

LIMIT,5

At this point we’re ready to switch over the station to interactive mode. We do that, by typing IAC into the station prompt:

Once initialization done, you’re screen goes black and all you’re left with is a prompt at the bottom of the screen. Next, you’ll have to logon – again – into the mainframe. You do that by typing /LOGON into the prompt (‘local’ commands are escaped with a leading / character):

Now that you’ve established a connection to the mainframe, you can start issuing commands. These commands are the same as they appear in JCL datasets. The first thing you have to do is to identify yourself (you’ve seen logon containing no account information). You do that with an ‘ACCOUNT’ statement. Something like this (all commands must be terminated by a ‘.’):

ACCOUNT,AC=CRAY,APW=XYZZY,UPW=QUASAR.

After this, we’re finally ready to transfer the image of the editor from the station (expander disk) to the mainframe:

FETCH,DN=TEDI,MF=AP,AC=ST,TEXT=BIN/TEDI.

What we’ve said here is to grab the dataset named as ‘BIN/TEDI’ (yes the IO subsystem did support directories) from the expander disk and make it available for the job (our interactive prompt) as TEDI.

At this point we can start the editor:

TEDI.

Eventually you’ll be greeted by this:

Here you have to enter a dataset name. It doesn’t matter too much what the name is (just don’t start it with a ‘$’). If it doesn’t exist, the editor will create a new one:

I’m going to stop hand-holding at this point, partly because I don’t know how to use TEDI either, and partly because it seems to have extensive help built in.

Before we move on, here’s a short video on interactive jobs:

Under the hood

If you’re still with me, you might be interested in what it took to get to this point. Maybe not. But that’s what I’m going to write about anyway, so you have no choice.

There are three important changes that made all the above possible

Bugfixes

Yes, there were bugs. Not too many, but enough to cause issues. They congregated (surprise surprise) in the so far not well-exercised vector instructions.

Watchpoints

I’ve resisted this idea for the longest time, but to track down the above mentioned bugs I needed a way to surgically enable or disable instruction tracing. Watchpoints trigger when a certain memory address is accessed and to implement them, I need to monitor every memory access the CPU makes. This I thought would slow the simulation considerably down but that was not the case – most likely because my simulation is already dog slow. Now that they are implemented, you can enable them in the configuration file, using the following syntax (on the global level):

1

2

3

4

<span>WatchPoints{

0x0387F8"Watchpoint 1"

0x01FE56"Watchpoint2

}</span>

The first entry is the (physical) address, the second is a string. When the specified address is accessed (read or written) an event gets fired with the following text:

By setting an event that matches this message you can turn on logging, or do other interesting things:

1

2

3

<span>EventPoints{

"Watchpoint at address 0x0387F8"{Type LogOn}

}</span>

Since the <hit count> section gets incremented for every hit (starting at 0), you can also implement counting, so for example you can wait until the 3rd access to turn on tracing.

Now that I think about it, this is probably how I should have implemented breakpoints as well. Oh, well, that will go on the todo pile…

Missing sectors

Remember back in the early days, when I’ve worked on recovering the disk image, I couldn’t successfully extract all the sectors? I’ve left that work off with 672 missing sectors. As it turns out, one of them – at least – was essential for job execution: the CPU calls into code that should be there. So I needed to re-visit the recovery effort and try to extract more sectors.

My original plan was to build on the voting idea that worked well in the previous effort and expand it: instead of using just one scan and ‘fold’ it over several times to line up the various reads of the same physical location, I could use several adjacent scans as well. That would greatly increase the number of samples I have for any particular part of the disk, increasing my chances for a good result. The trouble of course lies in finding the corresponding pieces in the different scans. My approach was that if a scan had at least one recovered sector, that creates an ‘anchor’. I could use these anchors to link the scans together and form a single unified scan. I can then fold them together, and easily get 10+ readings for every original bit.

Cool idea, but did it work? Not a bit. In fact, when I’ve printed the bit-length histograms, they looked horrible. Instead of the peaks getting better, they got almost completely washed out. So what went wrong? Maybe I didn’t do the alignment correctly? After all one bit offset in the wrong direction can affect the histograms a great deal. So, how to find the perfect alignment? I’ve implemented a (binary) correlation function – essentially calculated the average number of differing bits. The smallest this number is, the better the match. So, all I have to do is to find the minimum on this function by varying the offset and voila, I have the best alignment possible. So did that work? Not at all. But now I at least had something to look at, namely the value of this correlation function at the minimum. Turns out it was around 0.1 in most cases. That means that on average one out of every ten bits was different even in the best match. Not a big surprise then that these efforts didn’t pan out: the different readings of the same parts of the disk were quite different.

Back to the drawing board, I started thinking about other ways to recover the sectors. It was clear to me that using multiple samples and statistics will not lead to anything decent, so how can I improve recovery of a single reading? One thing I’ve tried was to make sure recovered sector numbers are sequential. If I’ve read sector 3, the next one should be sector 4. Unless sector 4 is unreadable of course. I honestly don’t fully understand why, but this caused a bunch of sectors to read properly, ones that didn’t before. My working hypothesis is that now the assumed sector starts – points in the bit-stream where I try to recover a sector – changed. Previously I couldn’t recover these sectors because I haven’t even tried. I would have to go back and validate this idea, and if turns out to be correct, see if I can get even better by altering the start point search even further. For now however it’s good enough: I have the missing sector for the boot image which allows for job execution. In fact, at this point I have no missing sectors in the boot images!

In closing

This is already a long article, so time to wrap it up. You can download the newest release from the download page. From here however the future is very uncertain. To gain the full functionality of the system at minimum I would need tools: COS, the assembler, CFT, the Fortran compiler, SEGLDR, the linker and the libraries: $CLIB, $FTLIB, $IOLIB, $PSCLIB, $RATLIB, $SCILIB, $SYSLIB, $UTLIB, maybe others too. I have some leads to some of them, but if you have anything, anything, please let me know. Without at least a subset of these tools and libraries, further work on the project is rather pointless. It would not lead to a more functional system, only a more polished version of the uselessness that it is today.

As an interesting side-note, the disk contains one dataset (JINSTALL) that would restore the full system from backup, trouble is, the tape that it would use to restore the system from is missing. If you look at the 3rd file in that dataset, you get a list of files that comprised the base system:

JOBCLASS

JGENCAT

GENCAT

JDIAGGO

$DIAGLB

LOCALS

STARTUP

STARTARCHIVE

CRAY1SYSTEMDUMP

JSYSDIR

$ADMLIB

$APTEXT

$ARLIB

$CLIB

$DBCTL

$DBHELP

$DBTXT

$DIAGLB

$FTLIB

$IOLIB

$LEXLB

$PDDERR

PROCLIB

$PSCLIB

$RATDEF

$RATLIB

$RATXVS

$SCILIB

$SID

$SYSDEF

$SYSLIB

$SYSTXT

$TDGTOC

$UTLDEF

$UTLIB

$UTLTXT

$WTOOLS

$YYPLB

ACCOUNT

ACCTDEF

ADSTAPE

APML

AUDIT

AUDPL

BACKUP

BIND

BLOCK

BUILD

BUPIO

BVCEDIT

CAL

CC

CFT

CFT77

CHARGES

CLEANUP

CLUPIO

COMPARE

CONNECT

COPYD

COPYF

COPYR

COPYU

COSDEF

COSTXT

CPP

CSIM

DDA

DEBUG

DRD

DSDUMP

DUMP

DUMPGOS

EXTRACT

FDUMP

FLODUMP

FTREF

ISP

ITEMIZE

JCSDEF

LDR

LD2

LOADGOS

MANAGE

MODSEQ

MODSET

MTDUMP

NEWCAL

NOTE

PASCAL

PDSDUMP

PDSLOAD

PERFMON

PREMULT

PRVDEF

QSTSUI

QUERY

RECALL

RECIO

SLSTACK.RELEASE

RELOAD

RESTORE

RETIRE

SLSTACK.REWIND

SEGLDR

SEGRLS

SETOWN

SKIPD

SKIPF

SKIPR

SKIPU

SKOL

SKOLREF

SKOLTXT

SORT

SPAWN

SPY

STATGOS

STATS

STOPGOS

STEP

SYSREF

TARGET

TDI

TDUMP

TEDI

TG

UNB

UNBLOCK

UPDATE

WRITEDS

EDITOR

This is going to be my wish-list for Santa this year.

To leave you on a higher note, there was one more thing I’ve uncovered: the line-printer attached to the expander supported a graphical mode. The banner, printed on every page used this mode. In the simulator – since text files don’t have a graphical mode – I’m using very simple bit-map graphics to print these: X for every dark pixel, and a space for every white one. Of course this way the graphics doesn’t line up with the text, but at least you can see it. So what was the graphics that was printed?

14 thoughts on “Jobs”

just to give you another link of possible software: I found on internet a document about a LISP compiler (PSL) for Cray X-MP, used by a well known mathematical software called REDUCE.
I contacted the person that did this version Winfried Neun but I didn’t get any reply yet.
This is the link to the document:

Dear Andras,
thank you very much for the great work you have done ! With your simulator you have created the possbility that work done in various fields of scientific computing during the era of the Cray X-MP could potentially be saved for historic purposes. However this will require Fortran compiler, assembler and libraries within COS on your simulator. I have found a website of former Cray employees: http://www.excray.com . May be one of them can help to find the missing tools you are looking for ?
Regards,
Juergen

For those who want to play around with the system, using the interactive mode: The extender disk attached to the IOP-0 station contains a program BIN/AUDIT, which is the equivalent of “dir” or “ls” (lists all datasets). I haven’t figured out yet how to list the filesystem of the extender disk, I’ve been looking at the hexdump. The AUDIT program doesn’t seem to work correctly yet; it displays various error messages, cannot show the permanent dataset names or any extra information using the LO option, even though this information is present on the Cray disks (verified by hexdump). But at least it shows an increasing number of datasets if you create new ones. Here’s how the calling sequence looks on the station:

You can create your own datasets with permanent names using the SAVE command. The commands of the Job Control Language are described in the CRAY OS Reference Manual, e.g. here:http://bitsavers.org/pdf/cray/COS/

Hello! As many already said, great job!
Because I find this very interesting to me, I’m curious is it possible to write program in assembler for Cray, then add that program to the disk image and then run it as job? I mean, it should be possible, but did you try something like that?

Hi! Thanks for the interest. No, I haven’t attempted to create my own assembly-based programs for several reasons:
– I don’t have an assembler, which means I would have to either write one, or hand-translate the instructions into their binary equivalent.
– Even if I did that, I don’t have any libraries, so doing anything meaningful, like accessing I/O or even printing ‘Hello world’ is next to impossible.
– Even if I somehow overcome that, the binary format of the executable images is undocumented (at least I don’t have the documentation) so execution inside the OS is still very hard.
The best option to execute your own code I think would be to take an existing binary (like the TEDI image) and patch it up to do what you want to. That way you don’t have to worry (too much) about the binary format, you have access to the (linked together) libraries, but of course you still have the problem of hand-translating your code to binary format.