Main menu

Tag Archives: ASM

With older generations of Mainframe Operating Systems, certainly MVS/XA and perhaps MVS/ESA, application performance tuning was a necessity, not an afterthought. Quite simply, the cost of Mainframe resources, namely CPU, memory and disk, dictated that your mission critical business application might not perform to business requirements, unless you tuned your programming code. Programmers, both of the system and application variety understood the bits and bytes of available programming languages (E.g. ASM, COBOL, PL/I) and Operating System (I.E. MVS), collaborating either via proactive process, or reactive problem solving. With the continuing reduction of IT hardware component costs, the improvement in Operating Systems (E.g. 64-bit architecture) and newer programming languages (E.g. C, C++), it seems that application performing tuning is somewhat of an afterthought, but at what cost?

We all know that the cost of a Mainframe MIPS is significant, and although it might have reduced dramatically from a hardware viewpoint, from a software viewpoint, the cost remains largely static at ~£1,500-£3,500, per year, depending on your configuration. So if your applications are burning several hundred if not several thousand extra MIPS unnecessarily, that’s very expensive indeed! Additionally and just as importantly, a badly tuned system will manifest itself in slower transaction response times and longer batch jobs, if applicable, which could impact service availability. So why is there a seeming reluctance to tune business applications, Mainframe resident or not?

If ever there was a functional IT area where the skills gap has never been wider, then application performance tuning is said skill, when comparing the salty old sea dog Mainframe dinosaur, with the newer Mainframe technician!

From an application development process viewpoint, where does the application performance tuning task live; before or after implementation? The cynical amongst us will know; if it’s after implementation, there’s a strong likelihood said activity will never be performed! If it’s before implementation, how many projects incorporate a meaningful stress test, or measure transaction response times versus an SLA or KPI metric? Additionally, if the project is high-priority and/or running behind schedule, then performance testing is an activity that is easily removed…

Back in the good old days, the late 1980’s to early 1990’s, some application performance tuning tools did start to emerge, most notably Strobe. Strobe was useful to even the most accomplished of system and application programmer personnel, and invaluable to less experienced personnel, and so arguably Strobe became the de facto software tool for tuning Mainframe applications. However, later releases of MVS (E.g. OS/390 and z/OS), the non-event that was the Year 2000 (Y2K), seemed to remove the focus on and importance of application tuning.

Arguably most importantly of all, that software MIPS cost item, where Strobe and its competitors (E.g. ASG/BMC TriTune, CA Application Tuner, IBM APA, Macro4 ExpeTune, et al) will utilize even more CPU to capture diagnostic trace information, contributed to the demise of application performance tuning. However, those companies that have undertaken such application tuning activities in the last decade or so are sitting pretty, having reduced the CPU (MIPS) resource consumed, lowering TCO and optimizing performance accordingly. In the 21st Century, these software solutions are classified as Application Performance Management (APM) solutions.

Is there a better and easier way to stimulate an interest in the application performance tuning discipline? If the desire exists to tune an application, lowering CPU MIPS usage, optimizing service performance, then the traditional tools and methods mentioned previously exist, but perhaps a new (or not so new) CPU performance data source exists…

With the introduction of the z10 server, a new function CPU MF (CPU Measurement Facility) was incorporated. Let’s not forget, z10 is now an n-2 technology, having been superseded by the z196/z114 and the latest zBC12/zEC12 generation of servers. So each and every committed Mainframe customer should be positioned to benefit from the CPU MF function.

CPU MF provides optional hardware assisted collections of information about logical CPU activity executed over a specified interval in selected Logical Partitions (LPARs). The CPU MF counters function is intended to be run on a constant basis to collect long-term performance data (I.E. SMF Record 113), in a similar manner to how you collect other performance data. I have previously briefly discussed how CPU MF SMF data can be used to increase Mainframe Server Capacity Planning efficiencies.

The CPU MF sampling function is a short duration, precise function that identifies where CPU resources are being used, to help you improve application efficiency. Put very simply, CPU MF sampling data has minimal CPU overhead (E.g. ~0.1-1.0%) when collecting data (I.E. z/OS Hardware Instrumentation Services – HIS), but this data can then be used to identify CPU “hot spots”, which can then be further analysed to identify the “areas of code” generating the high CPU usage. However, it was forever thus, whether an APM tool, or CPU MF sampling data, high CPU usage can be identified, but the application programmer must undertake the task of optimizing the application code!

IBM have done a great job in providing CPU MF counters data, optimizing the Capacity Planning process with the SMF 113 record, and the realm of possibility exists with the sample data, but a software solution is required to analyse and summarize this data.

Currently there are very few if only one software solution that analyses CPU MF sample data, namely zHISR from Phoenix Software International. zHISR interfaces directly with z/OS Hardware Instrumentation Services to collect data for hotspot analysis of customer, vendor, or operating system program execution. zHISR features include:

Support for up to 128 simultaneous data collections events. zHISR collections do not interfere with any HIS functions including sample or counter collection.

System console commands for many zHISR functions.

An Application Programming Interface to COBOL and Assembler for starting and stopping data collections. Collection lengths for API generated collections have a time range of one second or more.

Ability to schedule a collection with JCL so that collection starts when a given job or step begins.

Ability to store data collections as z/OS data sets or UNIX files.

Support for collections against CICS/TS transactions.

Analysis based on a time range within the collected data for a narrower spotlight on problem code.

An intuitive ISPF dialog allows the user to easily produce a CPU hot spots analysis, which can then be used for identifying the offending code sections. The user can then drill down and highlight the high CPU CSECT and program offset (instruction), comparing with their Associated Data (ADATA), and thus the source programming instruction. Therefore the skill required to perform analysis is minimal, as is the CPU overhead in collecting analysis data, and so eradicating the potential barriers when embarking on an application tuning initiative. Furthermore, the actual cost of deploying the zHISR software is not onerous and so perhaps each and every committed Mainframe user can easily include application performance tuning into their application development lifecycle processes.

zHISR has a UNIX file system interface that lets you navigate the system and browse or delete files. With zHISR, users can start and stop hardware event data collections and view the status of the current or prior HIS run. zHISR also includes a memory display/alter utility that lets you view main storage in the CPU you are logged on to. If zIIPs are present and zHISR is defined as an authorized subsystem, nearly all of the CPU processing used by zHISR is redirected to a zIIP.

There are also instances, however few and far between, where Mainframe customers have written their own proprietary in-house OLTP (On-Line Transaction Processor) and Relational Database Management Subsystem (RDBMS), where traditional APM software tools can’t provide a solution, only interfacing with underlying subsystems (E.g. Adabas, CICS, DB2, IDMS, WebSphere, et al). In these instances, CPU MF and zHISR offer a solution to help such customers, who probably face challenges when they upgrade their Mainframe servers, safeguarding software and application code is compatible with the new hardware, and ideally, exploits the latest functionality.

In conclusion, application performance tuning has to be a very important if not mandatory activity for the Mainframe Data Centre. Whether via CPU MF or traditional APM software solutions, the cost reduction and performance improvement benefits of tuning should be compelling reasons to proactively engage in application tuning activities. From a skills viewpoint, maybe the KISS (Keep It Simple Stupid) principle can apply, where CPU MF collects the data very simply and efficiently, complemented by zHISR, analysing the data in an intuitive and cost optimized manner.

It’s just not science fiction films that get rebooted or reimagined, the same thing happens in the technology world, although not always as obviously. Over the years we have seen Star Trek, Star Wars, and numerous superhero stories be updated for current day audiences, some better than others, and similarly, flash memory or Solid State Drives (SSD) is not a new concept in the Mainframe world. For me, I wonder whether the Back To The Future films might be rebooted, and if they do, hopefully it’s done well! Anyway, I digress, so back to the Mainframe flash memory observations…

Flash Express is a new feature for the zEC12 server, designed to help drive System availability and performance to even higher levels.

Flash Express cards are delivered as a RAID 10 mirrored pair for superior resiliency and reliability. In the unlikely event of device failure, Flash Express cards can also be concurrently replaced. The cards are designed for superior wear levelling and have a long expected lifetime. From a security perspective, Flash Express stored data remains protected. Data is encrypted on the Flash Express adapter with 128-bit AES encryption. Encryption keys are stored on smart cards that are plugged into the SE. Removing the smart cards renders the data on the card inaccessible.

In the first instance, IBM seems to be targeting Systems Paging (I.E. Auxiliary Storage Manager – ASM) and System Dumps (E.g. SVC, Standalone) as an introduction to this new level of storage hierarchy.

Flash Express reduces latency for critical system paging that might otherwise impact the availability and performance. For system paging flexibility and efficiency, Flash Express is a higher performance option when compared with traditional auxiliary storage (I.E. Disk). z/OS uses both Flash Express and page data sets for auxiliary storage by paging data to the preferred storage medium first, based on response times, data set characteristics, and other parameters. Wherever possible, the system will page first to Flash Express resulting in faster performance. Especially for data intensive applications the use of Pageable Large Pages with Flash Express enables the transfer of large amounts of data at faster speeds, which can result in improved performance for DB2 analytic workloads.

NB. Because Flash Express is not persistent across IPL events, it cannot be used for Virtual I/O or PLPA data used in warm starts. VIO and PLPA datasets must still be defined on DASD.

During diagnostic collection, as in SVC or standalone dumps, IBM states that systems can become sluggish effectively rendering key systems unavailable. When data is transferred into main memory as part of a dump event, the fast I/O rates and low latency associated with Flash Express provide decreased first failure data capture time, and faster page-ins of the critical pages needed for dump creation. This allows the system to return to normal workload performance faster, without incurring extra delays.

So clearly, adding a flash memory layer to the Mainframe storage hierarchy can only deliver benefit, and over time, seemingly Flash Express can be used for other I/O intensive applications, further improving response times and decreasing associated CPU cycles.

No doubt the non-Mainframe technician might say “so what, we’ve been doing flash and SSD for nearly 20 years. Typical Mainframe, always behind the rest of the IT world”!

Hmmm, perhaps not, hence the Back To The Future reference. Depending on your viewpoint, and perhaps urban myth, from a release date viewpoint, but in the mid-to-late 1980’s, StorageTek introduced a device called the 4080, AKA Control Data Set Manager (CDSM) for their Host Software Component (HSC) product. You will still see this 4080 device referenced in the HSC Systems Programmer’s Guide technical manual. The 4080 is a Solid State Device, utilizing memory storage, emulating 3380 and 3390 device types, allowing z/OS (MVS) data sets to be created and accessed, as and if required. Arguably if not certainly, this was the first instance of logical and physical device separation for Mainframe disks, and urban myth might dictate that this StorageTek 4080 technology was used or at least borrowed for the first EMC Symmetrix disk subsystem…

StorageTek typically packaged this 4080 device within a NearLine (ATL) solution sale and customers used this SSD device for HSC Control Data Sets, but also other high I/O files, such as the IMS Write-Ahead Data Set (WADS). Therefore, Flash Express isn’t the first SSD solution for the Mainframe, and similarly, Flash Memory/Cards, USB Sticks, et al, aren’t the first SSD type technology in the IT world.

As somebody (Machiavelli) far wiser than I once said “whoever wishes to foresee the future must consult the past”. The longevity of the Mainframe, nearly 50 Years of technical innovation, largely dictates that technological ideas are at least borrowed from the Mainframe, whether System Paging, Fibre Channels, Storage Area Networking (SAN), Virtual Storage, Flash/SSD Memory, naming but a few.

Should today’s zEC12 Mainframe customer deploy Flash Express? Without doubt, but to deliver maximum ROI and benefit, it should be considered as more than a faster paging and dump solution, and we look forward to how IBM will further enhance this product offering.