CMS/TSO PipeLines for Windows - Part III User Stages

Introduction

CMS/TSO PipeLines is a product that runs on IBM mainframes. On a z/VM system it runs uner CMS. On
a z/OS system it runs under TSO. As the name suggests data is passed thru a series of builtin or
user defined stages. Along the way the data can be modified or deleted from the pipe. You may be
saying to yourself there is nothing new in that. What is different is that PipeLines supports
concurrent parallel pipes and data can be routed into and out of a pipe.

System Requirements

While the PipeLib class library can be accessed from a .Net program, I think you will find
that running it under PowerShell more useful. I do all of my testing through PowerShell and
any pipelines I supply will most likely be PowerShell scripts. Therefor as minimum system requirement's
I recommend the following.PowerShell V1.0
.Net Framework V2

Invoking a User Stage

A user stage is invoked by the PS built in stage. This stage takes as
parameters the name of the user written stage and any parameters the user wishes to pass
to the user stage. The basic outline of a user stage is as follows:

Line 1 is the only line that is mandatory. It must be present in every script that uses
any of the user stage commands. Lines 3-8 sets up the basic data flow needed to keep the data
records in sequence. This is not enforced and the user is free code what ever is best for the
application.

The $usp variable defined in line 1 is the interface that connects the script to the
PipeLib class library. The pipe user commands check for the presence and the
validity of this variable.

Pipe User Commands

Below is the list of commands available in a pipe user stage.

Run-CallPipe

This command executes a pipeline in parallel with
the current pipeline. The user stage is suspended while the new pipeline runs. The user stage resumes
when the CallPipe command ends. The string that defines the new pipeline is passed as an
argument to this command.Examples: run-callpipe $pipe or run-callpipe "pipe definition"

Run-AddPipe

This command executes a pipeline in parallel with
the current pipeline. Unlike the CallPipe command the user stage continues to run. The string
that defines the pipeline is passed as an argument to this command.Examples: run-addpipe $pipe or run-addpipe "pipe definition"

Usp-PeekTo

This command reads without consuming the next record passed to the
stage. The name of a variable to receive the record is passed as an argument.Examples: usp-peekto "varname"

Usp-ReadTo

This command reads and consumes the next record passed to the
stage. The name of a variable to receive the record is an optional argument.Examples: usp-readto "varname" or usp-readto

Usp-Output

This command writes data to the currently selected output stream.
The output data is passed in a variable or a literal string.Examples: usp-output $line or usp-output "literal data string"

Usp-Select

This command selects the data stream to used for input and/or output.
The primary input and output are selected by default when a user stage is started.
This command requires two parameters:

This command shorts the currently selected input
stream to the currently selected output stream.
Examples: usp-short

Usp-Sever

This command severs the currently selected input or
output stream. The single parameter specifies which stream to sever. If the severed stream was
redirected by a CallPipe or AddPipe connector, Then an attempt is made to restore the
original connection.
Examples: usp-sever "input" or usp-sever "output"

Usp-StreamNum

This command gets the stream number of the
specified stream. The command takes two parameters:

The type of stream to process: "Input" or "Output"

Stream number or identifier. This parameter is optional. If omitted the currently slected input
or output stream is used.

If the stream is connected the $rc variable will contain the stream number. If not connected it will contain a -4.
Examples: usp-streamnum "input" 1, usp-streamnum "output" "id"

Usp-StreamState

This command queries the status of the specified
stream. The command takes one or two parameters:

When the first parameter is "Input" or "Output", the second parameter, if specified, is the number,
or id, of the stream to query. The default is the currently selected stream.

When the first parameter is "Summary", a return code of 0 indicates that at least one input and
one output stream are connected.

When the first parameter is "All" then the second parameter is the name of variable to recieve
the status of each pair of streams.

Examples: usp-streamstate "input" 1, usp-streamstate "all" "varname"

Usp-MaxStream

This command returns the max stream number for
the selected stream type The command takes one parameters:

The type of stream to process: "Input" or "Output"

Examples: usp-maxstream "input", usp-maxstream "output"

Usp-AddStream

This command adds an unconnected stream to the
user stage. The command takes two parameters:

The type of stream to process: "Input", "Output", or "Both"

Stream identifier. This parameter is optional. If omitted then use Usp-MaxStream to get the number
of the added stream.

Examples: usp-addstream "input" "strmid", usp-addstream "both"

AddPipe and CallPipe

Of the above commands CallPipe and AddPipe require special attention. These commands
insert a parallel pipeline into the existing pipe line and depending on the options coded for parallel
pipe it may or may not interact with the existing pipeline. The pipe definition syntax for the
CallPipe and AddPipe commands contain one additional element over that of the
Run-Pipe command.

Specify the character to be used to separate the stages of a pipe. It can be specified
as a single character or as the hex digits of a character. The default separator is the
virtical bar, "|".

ENDchar xorc

Specifiy the character that separates the pipes of a pipeline. It can be specified
as a single character or as the hex digits of a character. There is no default.

ESCape xorc

Define an escape character that can be used to override the processing of a
character that has special meaning to the PipeLine parser. Any character that follows the
escape character is treated as an ordinary character. There is no default.

NAME pipename

Specify a name to be associated with this pipeline.

TRACE

Turn on tracing for the pipeline.

Pipeline

stagesep

The stages of a pipe are separated by the STAGESEP character. The default is "|".

endchar

If there are multiply pipes in the pipeline, separate them by the ENDCHAR.

NO

If tracing is turned on for the pipeline then "NO TRACE" will turn it off for this
stage of the pipeline.

TRACE

Turn tracing on for this stage of a pipeline. If prefixed by the "NO" keyword then
tracing is turned off for this stage.

stagename

The name of the stage to be executed.

stageoptions

Supply any options needed by the specified stagename.

*

Identifies this as the start of a connector definition.

INPUT

This connector applies to an input stream of the user stage. If the connector is the
first stage of a pipe then the user stages input stream is directed to the first stage
of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then
the output of the CallPipe or AddPipe pipeline is directed to the user stage input.

OUTPUT

This connector applies to an output stream of the user stage. If the connector is the
first stage of a pipe then the user stages output stream is directed to the first stage
of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then
the output of the CallPipe or AddPipe pipeline is directed to the user stage output.

*

This connector applies to the currently selected input or output stream.

streamnum

The stream number of the effect user stage data stream.

The stream ID of the effected user stage data stream.

:

Marks the end of a connector definition.

label:
label.streamid:

A label that identifies additional input and/or output data streams for the stage it is specified on. The first occurrence of a label is called a label definition. It establishes connection point where other stages, in other pipes, receive data from or send data to. Each subsequent use of the same label is called a label reference. Use label references to define additional input and output streams for the stage. To use a label reference, specify a stage containing only a label with no stagename.

A streamid assigns a symbolic name to a stream. Name a stream by adding an identifier to the label. Write the label immediately followed by a period (.) and up to 4 alphabetic characters or a combination of alphabetic characters and digits that includes at least one alphabetic character. A stream identifier must be immediately followed by a colon with no intervening blanks.

The Connector element is used to redirect the data streams of the user stage to the pipeline defined by the CallPipe or AddPipe.

Consider the pipe "A|Z|D" where "Z" is a user stage that issues a CallPipe or AddPipe.
The following demostrates the effect of connectors on data flow through a pipeline when CallPipe is used.

Initiate the pararell pipe. The output of this pipe is directed to the
primary input of this stage. The previous input stream is svaed for a posible
reconnection later.

This stops the user stage from doing an Auto Commit. All stages start at some commit level. For user stages it is -1. Most of the builtin stages start at some level, verify the state of the stage and then commit to level 0. Stages at the lowest commit level run before stages at a higher commit level. All builtin stages will eventually end up at commit level 0. Thus with this command this stage will stay at level -1 until it explicitly commits to a higher level. Normally any IO request by a user stage causes an Auto Commit to level 0.

Read the first input record. Note that this read will consume the record. There is no need to worry about record sequence because the records never leave this stage.

After all records have been processed we restore the original input stream.

Here we commit to level 0. The other stages will have already commited to level 0 and are waiting for this stage to join them. Once all stages are at level 0 they are all eligible to run.

Define a pipeline to be executed.

Execute the pipeline.

Another Example

The following example of a user stage takes records that conatin binary data and dumps them
in hex format to the screen.

About the Author

Comments and Discussions

What is the current status, has there been any releases since the original. Anything new for ooREXX 4.0 ?? Also, the help for TCPCLIENT shows the help for UNIQUE, and TCPCLIENT does not seem to work like the CMS one, especially since i takes no options like oneresponse or greeting. Can you point me to anything to look at or read ??

There have been several bug fixes and new built in stages added.
Also there is an interface to ooRexx 4.0 for Windows and Linux.
On Linux Mono 2.4 is required. The problems with the html doc
will have to wait until the next release.