Links

Tags

z/TPFDF PUT 10 Enhancements

Several enhancements to the z/TPFDF product are available and included with z/TPFDF PUT 10.

PM72950 - Support files with over 100 million fixed file ordinals

When the z/TPFDF utilities were first provided, customer databases did not approach having 100 million fixed file ordinals. However, database sizes have grown over the years, and several z/TPFDF output messages and displays did not support nine digit ordinal numbers. If a z/TPFDF file would be defined with an end ordinal number that is nine significant digits, several z/TPFDF output messages and displays would truncate the first digit since they only supported up to eight significant digits.

With APAR PM72950, various ZFCRU and ZUDFM commands have been adjusted so that these utilities properly handle nine digit ordinal numbers (that is, they can handle up to 1 billion fixed file ordinals in a file). In addition, z/TPF APAR PJ40632 has adjusted the z/TPF recoup status output to properly display ordinal values with 9 significant digits. PJ40632 does NOT need to be installed in order for PM72950 to be installed.

PM81536 - Validate record IDs upon release and improve OPR-DB0100

With the ID parameter on the z/TPF RELFC macro, applications have the ability to specify a verification record ID that is compared to the ID in the record to be released. If the record IDs do not match, a system error is issued and the return is not processed. z/TPFDF is a middleware package that should provide such functionality. That is, it should ensure the correct file ID exists in a pool record before it releases that pool file address. In most parts of z/TPFDF, the block has already been accessed and its ID has been verified before z/TPFDF issues a RELFC on its file address. Therefore, an additional ID check on the RELFC API that is used internally is not needed in those cases. However, there are some routines in large logical record (LLR) and B+Tree processing where file addresses were released without accessing them first.

With APAR PM81536, these routines have been enhanced to validate the file ID before releasing the records in case those records are in use by another database. z/TPFDF LLR release processing has been updated to issue a find and wait on LLIBs and LLDBs to ensure the file ID is valid before releasing the blocks either directly or asynchronously. If an error is encountered, informational dump OPR-DB0189 is issued and the block is not released. The dump is issued only once per LLIB, but the loop to release additional file addresses continues. B+Tree processing has been updated to issue a find and wait on B+Tree nodes to ensure the file ID is valid before releasing the blocks. If an error is encountered, an OPR-DB0189 is issued and the block is not released.

In addition, a DB0100 system error is generated when an I/O error occurs when z/TPFDF attempts to retrieve a record. However, the online message did not indicate which I/O error occurred. It would be helpful for the online message to indicate which specific I/O error occurred, so z/TPFDF error processing has been updated with APAR PM81536 to make this change.

z/TPFDF data collection counters were originally designed to track the calls to specific z/TPFDF central database routines. Each of these routines had a specific task, such as obtaining a core block or releasing a record. These counts generally reflected the number of calls to z/TPF macros, such as GETCC and RELFC. However, this approach resulted in several inaccuracies. For example, when a DBDEL ALL,NOKEY or DBCLS RELFC is issued in the application while z/TPFDF data collection is running, the DFLNK START macro at the beginning of the central database routine that releases the records and forward chains would increase the RELFC count by one, but multiple RELFC calls would potentially be made in the routine. Similarly, many other z/TPF macro calls within the central database routines were not counted, or the data collection counters would be unconditionally updated when a routine was called even though the actual macro was conditionally skipped in the routine.

In addition, all data collection rates (per second) reported were approximately 5% higher than the actual values. This is because the data collection routines used the leftmost fullword of the TOD clock value for the start and stop time to calculate the run time. In reality, 1.048576 seconds would pass for every second counted. For example, if data collection ran for exactly 5 minutes, the internal routines would calculate the runtime as 286 seconds instead of 300 seconds. This inaccuracy resulted in the data collection "per second" rates being about 5% too high. Even though this discrepancy was consistent across all rates, the numbers reported should be accurate.

APAR PM77843 improves the accuracy of several z/TPF macro counts in z/TPFDF data collection by updating the z/TPFDF central database routines. For example, all occurrences of CRUSA macros were replaced with a LEVTA call followed by a RELCC call to ensure that the true number of RELCC macro calls are accurate. Similarly, rlcha() calls in B+Tree and LLR routines were replaced with a find/relfc/relcc loop to be able to count the actual number of relfc calls. All of these routines can now modify multiple data collection counters in the heart of the code processing rather than simply at the beginning of the routine. Finally, data collection segments were updated to more accurately calculate the data collection run time.

When running a z/TPFDF data collection after the APAR is loaded, you may notice that several data collection counts may experience (potentially significantly) different values compared to a similar data collection taken before the APAR is loaded.

The following counts may be higher or lower, depending on the application profile:

GETFC

The following counts may be lower because some of the application programming interface (API) calls previously included in these counts are now included in one of the previously listed counts:

FILNC, FILEC

This should be taken into account when you compare results from a data collection run that was done before this APAR is applied to results from a run that was done after this APAR is applied.

PM83026 - C API improvements

APAR PM83026 provides new functionality in the z/TPFDF C APIs in the following areas: T-type LRECs and key lists.

T-Type LRECs

z/TPFDF assembler applications can add, read, and delete T-type LRECs without directly manipulating the underlying W-type (work) file. However, corresponding APIs were not available for use by C/C++ applications.

To address this requirement, three new C/C++ APIs have been provided:

dfadd_ttype() – adds T-type LRECs to an underlying W-type file

dfdel_ttype() - deletes T-type LRECs from an underlying W-type file

dfred_ttype() – reads T-type LRECs from an underlying W-type file.

Using APIs to work directly with t-type files in C/C++ applications provides the following benefits:

Applications do not need to manage the underlying W-type files. In particular, that means that separate database definitions (DBDEFs) are not needed, and applications do not need to independently open and close the underlying W-type file or manage references to the SW00SR.

Applications written in both assembler and C/C++ that need to access the same working storage can more easily co-exist.

By using t-type support in C/C++, applications don't need to manage the allocation and de-allocation of temporary work storage.

Key Lists

A C/C++ application can set up a key list with a search argument value in a variable and then call dfkey() to "activate" the key list. This process stores the address of the search argument value in the SW00SR. The benefit to this approach is that the contents of the variable can be altered and will be automatically referenced on subsequent z/TPFDF API calls without requiring dfkey() to be repeatedly called. However, using this approach, if the search argument variable falls out of scope (for example, it is released or is used outside of the function where it is defined), a subsequent z/TPFDF API call will potentially encounter addressability issues when it attempts to access the variable.

APAR PM83026 enhances the key list facility by providing two new C/C++ APIs have been provided:

dfkey_save()

dfkey_nbr_save().

These APIs save the actual search argument values instead of saving a reference to (address of) the key search argument variable. This allows applications, if they choose to use these APIs, to continue to use key search argument values without needing to be concerned with where the search argument variable was defined. The APIs take the same parameters as the dfkey() and dfkey_nbr() APIs, respectively.

PM88915 - Write z/TPFDF to file system

A business need exists to take data that resides in z/TPFDF databases and make that data available for processing outside of a z/TPFDF environment, or even outside of a z/TPF environment.

APAR PM88915 addresses this need by introducing new z/TPFDF APIs that read data from an existing z/TPFDF database and write the data to the z/TPF file system.

One new assembler API and two new C/C++ APIs have been provided:

DBFSYS WRITE

dffsys_write_fname()

dffsys_write_fstream()

These APIs are similar to the existing DBDSP and dfdsp APIs in that keys are optionally provided and then z/TPFDF processing will select potentially multiple or all LRECs in a subfile to be processed on a single API call. The biggest difference is that instead of simply sending the data to the console, z/TPFDF writes the output to a file system file, which gives the user many more options for processing the data.

Several other parameters can be provided to customize how the data is processed. This includes:

Choosing whether the application is responsible for opening and closing the file system file, or allowing z/TPFDF to open and close the file on each API call.

Opening files in append or overwrite mode.

Having z/TPFDF put the length of each LREC in a 4-byte field that precedes the actual LREC.

Specifying a starting displacement within each LREC (or user LREC) to begin processing (effectively stripping the specified number of characters off of the beginning of the LREC or user LREC) and a maximum number of bytes to process within each LREC (or user LREC).

Providing a 1-byte character that acts as a separator between LRECs

Processing from beginning of a subfile or from the current location within a subfile

Optionally processing all subfiles within the z/TPFDF file.the current locat

For example, an application could specify that only certain information for all LRECs for a particular customer be copied from a z/TPFDF subfile to the file system, with each LREC separated by a comma.

Several new structured programming macro (SPM) equates (such as DBFSYS_OK) and C/C++ error checking functions (such as DFFSYS_OK) are also provided to interrogate the new return codes, which are set in SW00RT5.

In addition, a new customizable equate (#DF_MAX_FSYS) has been introduced in macro ACPDBE that allows you to limit the size (in bytes) of the file written to the file system.

PM89556 - SDO Metadata Enhancements

This APAR provides various enhancements to service data objects (SDO) metadata processing.

When SDO Access to z/TPFDF was first provided with APAR PK60030, the ZUDFM METADATA command with the VALIDATE parameter specified would only show one error message at a time because it stopped validation as soon as an error was detected. This would result in a time-consuming process of eliminating errors from the metadata, as the user would have to repeatedly make the correction, upload the corrected metadata, and run ZUDFM METADATA VALIDATE again.

APAR PM89556 improves efficiency in preparing the metadata by updating ZUDFM METADATA VALIDATE processing to validate as much of the metadata as possible and producing a report of all error conditions encountered. With this change, multiple reasons may be shown as part of message UDFM0532E, and new message UDFM0541E is provided when validation processing has completed with errors. Message UDFM0532E is changed to additionally show the number of errors detected in the validate process.

z/TPFDF DSECTs can contain labels and equates that are up to 63 characters. However, macro label set (MLS) was only designed to accommodate labels and equate names up to 20 characters in length. That is, ZUDFM MLS processing would truncate all label and equate names after 20 characters when storing these names in the MLS database (GR3NSR). This data is used during ZUDFM METADATA CREATE processing to create metadata for a database. As a result, the field names and default aliases in the metadata would be at most 20 characters in length. When attempting to use this metadata in ZUDFM METADATA VALIDATE processing, UDFM0532E METADATA VALIDATION FAILED messages result with REASON: DUPLICATE KEYORDER FOUND or REASON: DUPLICATE FIELD ALIAS messages would be generated. The user would then have to manually change the field aliases. This potentially unnecessary work would not be needed if the MLS database accepted labels up to the maximum length allowed.

Similarly, if a DSECT contained storage areas without labels, an entry would be created for each of these areas in the MLS database. When the metadata was created, entries would be created with a field name and alias of "" (no character between quotes). These are not allowed in the final metadata and thus would need to be manually adjusted.

With APAR PM89556, the ZUDFM MLS DSECT (GR3NSR) was updated to create an additional variable length field to allow labels of greater than 20 characters. ZUDFM MLS processing now uses the new variable length field if the label name is 20 or more characters in length. The field name in GR3NSR will always have a minimum of 20 characters. ZUDFM METADATA CREATE processing has been updated to use the entire label name instead of the first 20 characters. Furthermore, ZUDFM METADATA CREATE processing was updated so that storage areas without labels are considered “spare” fields that are ignored. This means that that they do not need to be manually adjusted to allow metadata validation to work.

With the initial release of SDO Access to z/TPFDF, the ZUDFM METADATA CREATE command created metadata in EBCDIC format and wrote the data in EBCDIC format to the z/TPF file system, even though the metadata itself specified encoding="iso-8859-1". In order for this metadata to be readable when it was FTP'ed to a client, it would have to be transferred in ASCII mode. This was not consistent with the traditional handling of XML files. For example, using the z/TPF Toolkit to open the metadata in the z/TPF file system or to copy/paste the metadata results in the Toolkit "Opening BINARY mode data connection" which ultimately led to an "empty file" error when attempting to access the file. The Toolkit Preferences for Remote Systems->Files could be changed so that *.xml files are transferred as "Text", but that would potentially result in other XML files not being handled properly. The correct solution would be for the SDO metadata files to be written to the z/TPF file system in Unicode UTF-8 format and handled in binary mode.

With this APAR, all XML that is built in EBCDIC format on ZUDFM METADATA CREATE and ZUDFC SDO STOP commands, respectively:
- specifies that it is in Unicode UTF-8 format
- is converted to Unicode UTF-8 format, and
- is written to the z/TPF file system in binary mode.
ZUDFM METADATA VALIDATE processing was updated to read the XML from the z/TPF file system in binary mode and to convert the XML from UTF-8 to EBCDIC format before using it.

z/TPF APAR PJ41246 is a co-requisite APAR for z/TPFDF APAR PM89556.

These APARs have some migration considerations.

For All Users:

After applying this APAR, you should enter the ZUDFM MLS command to keep the MLS database current with z/TPFDF system file information. Existing MLS information can still be used with this APAR applied, and any MLS information generated with this APAR applied can still be used if this APAR is fallen back.

For SDO Users

If you have DSECTs with labels greater than 20 characters and want to create new metadata for those DSECTs, make sure you enter the ZUDFM MLS command to rebuild MLS information for your database before entering the ZUDFM METADATA CREATE command.

The metadata created by the ZUDFM METADATA CREATE and ZUDFC SDO STOP commands is now saved in the z/TPF file system in UTF-8 format instead of EBCDIC. This metadata should now be FTP'ed off of z/TPF (and transferred back to z/TPF, as necessary) without performing any translation (that is, binary mode should be used instead of ASCII mode). If using the z/TPF Loader to load the metadata to the online file system, the ATOE parameter should not be used. If the wrong transfer mode is used, a ZUDFM METADATA VALIDATE command will detect the problem by generating a message similar to the following:

Any metadata that is stored on the z/TPF system that was created before this APAR is applied will not be able to be used with SDO applications or ZUDFM commands. You must re-load the metadata in the correct format to the z/TPF system. Attempting to use existing metadata with an SDO application will result in an error similar to the following:

DFED1002E 15.18.24 readData() CALL FAILED

ERROR=class com.ibm.tpf.dfdas.exception.MetadataException

TEXT=Error converting metadata to EBCDIC

If you have existing metadata that was generated without this APAR and FTPed off of z/TPF in ASCII mode, you can still use this metadata on the z/TPF system by transferring it back to z/TPF in binary mode.

If you use the z/TPF Toolkit to view the metadata files online in the z/TPF file system, make sure that the *.xml file type for Remote Systems Files has a File Transfer Mode set to XML or Binary.

If you need to fallback this APAR, any metadata that was created while this APAR was loaded can still be used if it is FTP'ed back to z/TPF in ASCII mode.

PM97115 - Improve z/TPFDF error handling if a device is not operational

When a device is not operational (that is, IDECSUD is set to equate CXSGNO, or x’88’, after a FIND macro), a DB0100 system error would be generated. If the application attempted to continue to access the device while it was not operational, streaming OPR-DB0100 system errors might occur because every access on that device would fail. In this situation, these system errors are not helpful. It would benefit system resource availability to suppress these system errors.

With APAR PM97115, z/TPFDF system error processing was updated so that in the DB0100 processing, if the value of IDECSUD is CXSGNO (x'88'), dump processing is skipped.