New Security Revelations: Governments Spying More than Expected

The New York Times, The Guardian, and Pro Publica are jointly reporting on new revelations about the extent of U.S. and U.K. (in particular) government surveillance of Internet communications. The revelations come primarily from U.K. GCHQ documents characterizing GCHQ and NSA capabilities. Former NSA contractor Edward Snowden obtained the documents and shared them with media outlets. Bruce Schneier, a security expert advising The Guardian, comments on the revelations and offers some practical advice.

I'm still absorbing the implications of these revelations. If they're true, I tend to agree with the security experts who are concerned about risks to people and their private information. One of the important roles of government is, ostensibly, to protect its citizens. If the government continues trying to undermine IT security in various ways, the government is making its own citizens easier to attack. Which is exactly backwards, of course: a security agency should be promoting the safety and security of its citizens, not undermining it. It doesn't take a Hollywood movie or even an Edward Snowden to understand that if the "good guys" can get in then so can lots of "bad guys." And the "good guys" have a lot more to lose when they're vulnerable.

I agree with Bruce Schneier that the IT engineering community will be doing a lot of work in this area over the coming weeks, months, and years to improve IT security and to better protect privacy. These revelations will also probably spur a lot of political discussion about the appropriate role of government and what the limitations on government should be. That's not a new debate, nor is it one that should ever end. In my view we must constantly remind ourselves of the Fourth Amendment to the U.S. Constitution, and we must meet or exceed that high standard.

OK, what about mainframes? Bruce Schneier's advice is heavily client (end point) focused, and that's appropriate for his readership. In the world of servers and enterprise computing there are also important considerations, and I would advise all IT professionals to pay close attention to security discussions and improvements coming out of the IT engineering community. I would also point out that I see way too much carelessness. I'm not talking about whether extremely well funded government intelligence agencies can access your applications and databases. I'm talking about rank amateurs. For example, do you have 3270 "green screen" terminal connections to your mainframe, for end users and/or for administrators? If yes, are those connections encrypted? You're sending mainframe user IDs and passwords across those links every day, across your wide area network perhaps. They're not encrypted, in the year 2013 (or even 2003)? Really? When exactly are you going to take security even half seriously?

As another example, is your idea of application integration to dump half your customers' most sensitive personal information into sequential files every night then FTP that — unencrypted of course — to dozens of different distributed servers, only to run a poorly secured application? How is that possibly secure? How is that being a responsible steward of your customers' private information? It isn't, yet I see it practically every day. Too many IT people think it's a good idea to copy data everywhere, all the time. There's no way you're ever going to protect your organization against even rank amateurs with that architectural approach. Stop copying data and start securing it. That means, paradoxically, opening up your mainframe to authenticated, authorized, and (usually) encrypted, direct access to application and information services. Why, just last week I had a conversation with an IT manager about this very issue. That manager questioned whether it was secure to access DB2 for z/OS directly from a PC-installed productivity tool. Compared to what? Compared to extracting all the data (not just the records the end user is supposed to be accessing) to a flat file, FTPing it (on a clear wire) to another database running on Microsoft Windows (!), then accessing it there, without any security context whatsoever? Of course that isn't secure. And I'm going to partially blame "mainframe people" — you know who you are — for setting arbitrary "security policies" which end users inevitably must circumvent in order to get their jobs done, or because they think they're "saving MIPS." I've even seen end user departments set up elaborate screen scraping tools on batteries of client PCs in order to perform data extracts, because that's what the "mainframe people" and their "security policies" require them to do to keep the business running. This madness must stop!

Now, for those two organizations in the world that have eliminated the low hanging vulnerabilities and that have stopped all the madness, I would recommend getting a mainframe if you don't already have one. (If you don't have one you probably aren't one of those two organizations.) Use your mainframe as your premier security hub to better protect your organization. We don't know everything yet — I'll keep reading the press reports with great interest — but what we do know from decades of experience to the present is that mainframes, well managed, have proven especially resistant to security threats. And, I write only half jokingly, we also know that the only organizations that might rival government intelligence agencies in their political power and influence are large financial institutions. All of them would presumably scream bloody murder if their core systems were exposed. Moreover, if you want open source software, you've got it on zEnterprise. Linux on zEnterprise is 100% open source software. There are no proprietary drivers or other closed source binaries required, unlike many other hardware platforms. z/OS has a large and growing collection of open source software available, too, and you can go grab whatever you like and quickly deploy it. (On z/TPF as well.) There's also the unparalleled statement of integrity for z/OS and for z/VM.

Canonical (Ubuntu) Needs a Mainframe: An Elaboration

I want to quote one of the comments Mark Shuttleworth wrote which I think illustrates his profound misunderstanding, a misunderstanding that might have contributed to Canonical's recent security failure:

Cloud architectures are fault-tolerant by software architecture, only an idiot would pay for the same fault tolerance twice. Therefore, no matter how hard IBM tries to sell the idea of mainframes being the hardware for cloud, I don’t see it panning out that way. The whole point of the work that’s brought us cloud is to be able to do very large, very reliable services on low-cost, unreliable hardware.

OK, forgetting for a moment that reliability is only one of the many qualities of service — security is another one, as Canonical has belatedly and tragically discovered — no, I disagree, and so do most IT professionals. The reason is very simple: everything in IT fails, particularly software and administration (people). Over the past few days I've repeatedly explained that IT doesn't work well unless you get both the hardware and the software right, and unless both are co-engineered to work together cooperatively, with really, really excellent, common, consistent autonomics that reduce the people risks as much as possible. That's especially true in availability engineering.

One of the many beautiful aspects of the zEnterprise family of solutions — IBM's decades-long genius, really — is that IBM always expects software to fail, whether its own software or its customers' software. Only last week I heard a long and painful story from a client. That client explained in great detail how often their pure software cluster failed, leaving thousands of users with nothing to do. Programmers are not perfect. Nor are hardware designers necessarily, but "defense in depth" is extremely valuable when engineering for high availability.

And that's what IBM has done and keeps doing. It's not that IBM hasn't tried other approaches. Decades ago IBM implemented software-based clustering in IMS, for example. It's merely "OK" by mainframe standards, meaning it's superb software clustering but it isn't what customers expect. IBM still supports that form of clustering, but a couple decades ago IBM introduced the first version of Parallel Sysplex which relies on a combination of common, hardware-based features and software-enabled products that exploit those hardware features. IMS is one of many examples but only one example. Parallel Sysplex evolved over the past two decades and continues to evolve and improve. (This month's announcement of RoCE memory-to-memory high performance networking is a good example. Ostensibly that's a hardware feature, and it is, but it's actually a clever, packaged, integrated combination that provides a common service to all applications, transparently. Always with multiple layers of availability and fault tolerance.)

Frankly it takes an amazing amount of hubris to suggest that programmers always get it right, every time, and never ever muck up what they previously got right. Or that it's not possible to learn from other engineers who took a different approach that actually works in the real world. Again, look at Apple. Why on earth did Apple buy hardware companies? Why do they have engineers who can design chips? Why are they reportedly investigating the purchase of their own chip foundry? Both hardware and software matter to achieve a particular business outcome. Granted, Apple isn't maniacally focused on maximum qualities of service enterprise IT engineering like IBM is with its zEnterprise solutions. Apple is engineering for different outcomes than IBM and literally never compete. But the core principle is the same.

I should say that I very much respect Mark Shuttleworth and his accomplishments. But I think he got this one wrong, very wrong. We all make mistakes sometimes, myself included. We hopefully learn from those mistakes, otherwise we're doomed to repeat them.

Canonical (Ubuntu) Needs a Mainframe (and zBC12 Post #3)

In my previous blog post I wrote "stay tuned," because I'd have a lot to write about the new software solutions IBM is announcing with the new zEnterprise BC12. In this post I'll start with some general observations on important foundational software capabilities that IBM is either improving or introducing for the first time. And as I've written many times (and as Canonical has perhaps realized), the marriage of hardware and all the software layers must be harmonious into order to achieve particular business outcomes and in order to deliver the best qualities of service. I'm also seeing IBM develop and ship new top-to-bottom solution packages which include zEnterprise server hardware and storage, operating systems, middleware, applications, maintenance, and services with simple, competitive, multi-year "no surprises" pricing. Inevitably there's always some customization involved in enterprise computing, but these zEnterprise solution kits greatly simplify the time-to-value.

Here are some examples of the new/improved foundational capabilities and associated solution packages:

IBM is introducing new hardware-based/software-exploited cryptographic algorithms, notably elliptic curve cryptography (ECC). IBM is seriously positioning zEnterprise as the enterprise security hub, with solutions for managing digital certificates across the enterprise, for example. The IBM Enterprise Key Management Foundation solution, introduced shortly after last year's zEC12 announcement, has been updated to include a zBC12 option and the new cryptographic algorithms.

IBM has formally announced z/OS 2.1. There's a lot to unpack in that announcement, and I might have to do that in a separate blog post, but one of my favorite areas of improvement is in z/OS UNIXTM System Services. As HP-UX and Solaris fade to black, IBM keeps improving z/OS UNIX as a warm and welcoming target environment for those workloads and for other applications. In particular IBM has adopted more of the new GNU conventions, and the z/OS C/C++ compiler picks up more of the latest C/C++ language standards, not to mention deeper exploitation of zEC12/zBC12 processor features for performance and scalability. Relatedly IBM has updated its Cognos business intelligence solutions to the latest version on z/OS so that customers can minimize data movement, keep their information most secure, and access reports from Web and mobile devices directly on zEnterprise. And if you want everything in one simple package, including a zEnterprise server, IBM enterprise storage, z/OS, DB2, the IBM DB2 Analytics Accelerator — with or without Cognos and SPSS — IBM's updated Smart Analytics System 9710 is perfectly packaged and ready to roll.

What about COBOL and PL/I which power, among other things, most of the world's large financial systems? IBM thinks its new Enterprise COBOL Version 5.1 is so attractive that it wants everyone to try it. Thus there's a new "developer trial" edition. You can order V5.1 and kick the tires without starting the 12 month single version charge (SVC) period. Enterprise COBOL V5.1 is the first COBOL compiler that can exploit specific processor models at compile time. IBM had to do a great deal of "plumbing" work to make this happen, and this version is but the opening salvo in a series of new COBOL compiler releases. IBMers on the forums are now openly discussing their COBOL roadmap to a 64-bit compiler, something they were hesitant to do until they got this new compiler technology in place. Go grab it. You'll like it. (And let's not forget the new PL/I V4.4.) But while a new compiler is an important foundational capability, IBM is doing a lot of work above that layer. A good example is IBM's Business Rules for z/OS which, in simple terms, means that application developers don't have to write or modify code to change business rules — those kinds of business changes can now be made much more easily in a graphical, code-free way. If you think the first and only way to solve a particular business problem is to program, think again. To borrow from Strunk and White, omit needless coding.

There are lots of new mainframe "freebies" — I'll have to update my list. Perhaps my favorite is the new z/OS Management Facility Version 2.1, and one of the big headlines there is that it's much easier to deploy and even less resource-intensive thanks to the new WebSphere Liberty Profile packaging. IBM is also letting everyone know that it expects to release a new Encryption Facility for z/OS that'll exploit the new zEDC hardware, so both compression and encryption will be turbocharged. zSecure Version 2.1 is also newly announced, to make security administration and auditing simpler and more reliable.

IBM is pushing its zEnterprise cloud credentials hard, and rightly so. The mainframe is the original and best cloud platform, particularly for secure private clouds. IBM announced z/VM Version 6.3 which now incorporates integration with the industry standard "OpenStack" initiative. You'll be able to manage z/VM-based cloud environments via OpenStack-compatible solutions. There are new z/VM features to reduce or eliminate planned outages of individual z/VM LPARs, notably "upgrade in place." Previously it was necessary to carve up system memory into 256 GB LPARs for z/VM, but now 1 TB z/VM LPARs are supported. (Memory scalability is often the limiting factor in cloud deployments, so this is an important increase.) Of course IBM has updated its Enterprise Linux Server (ELS) solution and application solutions based on ELS to incorporate these new capabilities.

That's just a sampling of core infrastructure capabilities and how they map to new and updated, packaged zEnterprise solutions. I expect to have more to say about this week's "announcement deluge" as I keep finding more and more gems, and I'll try to provide some more perspective on trends and directions.

EMC VMware Needs a Mainframe

For the record, I think VMware is a reasonably good product for what it does — although there are plenty of fine X86 virtualization solutions, like KVM — but mainframes are different and special. It's the combination of hardware and software, focused by design on the same goals and outcomes, that matters. (See also: Apple.)

Utah Department of Health Needs a Mainframe

Hackers, perhaps from Eastern Europe, stole the personal details of over 24,000 of Utah's Medicaid beneficiaries from a "server" operated by the Utah Department of Technology Services.

DTS has an "interesting" perspective: "The health department uses 125 of the state’s 520 servers.... 'We pride ourselves on our lean government.'" Those numbers are certainly not "lean" if you have mainframes. Raise your hand if you have 520 or even 125 mainframes. Oh, and I seriously doubt the State of Utah has 520 servers. Those are only the ones DTS knows about, and maybe not even that. For example, I doubt it includes servers at Utah's public universities.

The Utah DTS spokesperson goes on to imply that the solution to secure all those "lean" servers is...to hire more IT staff.

My Mainframe-Related Pet Peeves

In no particular order:

"Green screens" are good enough. No, they're not. Do you force your users to submit their input and receive their output via punched cards? User interfaces change and evolve, and appearance often matters. Stanford's IBM mainframe served the world's first interactive Web application. If you haven't provided Web user interfaces on your mainframe to serve users' demands, what on earth are you waiting for?

Everyone must use Web interfaces. Some users prefer to continue with their familiar, fast, and efficient 3270 terminal user interfaces. Let them coexist. One size does not fit all.

FTP overuse. FTP is not an application integration solution! Connect two applications using FTP and you've automatically converted two or more business process steps into a "we might get around to it, eventually, if you're lucky" business process. Do you think your customers want that? And why are you copying all that sensitive data anyway? To make it easier for bad guys to get?

We don't allow TCP/IP connections to our mainframe "for security reasons." Congratulations, that "security" policy inevitably leads to the least possible secure environment you can imagine as the business finds every possible workaround to keep doing business — a true security nightmare. Let the z/OS Security Server and RACF do their jobs, please.

"Open" platforms and storage. If you connect exactly the same storage unit on your SAN (that you're already using for everything else) to a z/VSE system in exactly the same way, does that suddenly make your storage unit "closed"? If you're one of the people responsible for typing in activation keys to make sure Microsoft Windows can actually function, are you the same person who thinks that z/OS and Linux on z, both which eschew keys, are "closed"? Words should have consistent meanings. Many IT vendors have thoroughly debased the word "open," and some of us have fallen for that particular word game. It's past time they stop — and that all of us wise up.

"Mainframes are expensive." You know what's expensive? Not knowing the value of your financial holdings during a financial crisis because you've scattered bits of your portfolio records into little servers — that's expensive. Letting unreleased Michael Jackson records escape before you can monetize them. Billions of dollars of credit card fraud. Building yet another massive data center. Paying for 60 more licenses of Brand O middleware (this week). Adding another 20 staff to your payroll (this week) to support the IT mess you've implemented. You know what's not expensive? Mainframes. Stuff that works well isn't expensive.

"But that would require us to add MIPS...." So what? Business growth is never free, but it's darn inexpensive if it's a mainframe that's growing. And do you see MIPS listed as a currency, next to the yen, dollar, euro, and pound? It's not. IBM has different prices for different workloads.

Mainframe chargeback regimes. Everybody does them wrong. It's only a question of how wrong. Just because a mainframe, as a standard feature, lets you count and apportion various technical quantities like CPU-seconds doesn't mean they have much cost accounting significance. You certainly shouldn't be putting prices on those technical quantities while everything else in your data center (and beyond) remains uncounted, nor should those prices be different than true marginal costs (which can often be zero or near-zero).

#2: You will never need a password again. Technically that's no problem whatsoever if you have a mainframe and hasn't been for many years. IBM has done a very good job preserving and extending the mainframe's leadership, positioning the mainframe as the definitive Enterprise Security Hub (or ESH if you like). For example, credit and debit card systems are already getting a lot smarter thanks in large part to the mainframe's security innovations. In an ever more interconnected era (see below) when security is becoming ever more important, more businesses and governments are turning to mainframe-based solutions. The only question in my view is whether mainframe professionals will lead or follow this trend. I vote for the former.

#4: The digital divide will cease to exist. Universal mobile access to computing is going to favor the mainframe. First, there's going to be a direct effect on transaction volumes in existing banking systems, to pick an example. I'm hearing lots of reports that's precisely what's happening, even with only a fraction of the world using smartphones at this point. Second, there will be heightened security requirements (see above). Third, the greater the audience depending on mobile access for services, the greater the cost of service interruptions, thus favoring more resilient systems and solutions. Fourth, the greater the demand, the greater the need for massively scalable systems, i.e. mainframes. That's due to the need for bigger central systems of record as well as worsening data center resource problems in procuring enough space, power, and cooling. The world's telcos, for example, are now seriously rethinking their entire infrastructure which is becoming too costly and unsupportable, after a couple decades of largely unrestrained build-out.

#5: Junk mail will become priority mail. I'm not so sure about e-mail, but the central point here is that transactions are becoming more complex, with more and more heavy information analytics associated with core business processes in order to tailor services much more precisely to customers. That's going to drive the need for massively scalable systems with tight integration. Sound familiar? IBM is right at the vanguard of that trend, with the DB2 Analytics Accelerator as a preeminent example. That technology alone is making whole new analysis-heavy applications possible that were simply never possible before.

What's your forecast? My immediate forecast (or at least wish) is for all of our readers to have a safe, healthy, prosperous, and happy new year.

SK Communications Needs a Mainframe

This report, which details the July intrusion into Korea's largest telecommunications company, an intrusion which resulted in unknown hackers collecting the personal details of up to 35 million Korean citizens, is absolutely chilling and horrifying.

Mitsubishi Needs a Mainframe (Updated)

Mitsubishi Heavy Industries (MHI), Japan's top defense contractor, has discovered targeted viruses on more than 80 of its servers and PCs. Japanese Defense Minister Yasuo Ichikawa is assuring the public that the viruses didn't transmit the weapons plans stored on those computers to another party, but the truth is that nobody knows for sure yet. The Japanese government has ordered MHI to undertake a full investigation, and ministers are quite angry MHI didn't report the incident much earlier. It's an extremely serious security breach.

The Chinese government denies that it was responsible for the viruses. However, most governments spy on other governments. There are other reported attacks targeting defense contractors and Japanese government agencies, but the perpetrator is still unknown.

I know a bit about MHI. MHI is a very big, complex company. Among its many subsidiaries and affiliated companies, MHI has some in-house information technology talent, and some of their people have strong IBM mainframe skills. Unfortunately, as evident in press accounts, MHI did not employ an IBM mainframe as its secure system of record for these weapon designs, which include designs for submarines, missiles, and destroyers. I hope that MHI will leverage its own mainframe-skilled people to support these high security requirements and other mission-critical applications. While the incident is quite bad, the important part now is to learn and to adapt — and to use the right server technology for the mission.

UPDATE #1: DigiNotar, the Dutch certificate authority (which reminds me of CardSystems), is now bankrupt. Hackers penetrated DigiNotar then generated signed SSL certificates which resembled authentic security certificates for Google, Facebook, Yahoo!, and other major Web sites. Hundreds of thousands of people, many in Iran, probably including many democratic movement leaders, then had their formerly secure Web browser sessions intercepted through so-called "man in the middle" attacks. Considering the results of a security audit, DigiNotar needed a mainframe. But it's too late for DigiNotar.

Meanwhile, over at The Coffee Bean shops in Singapore, there's a buy one get one free promotion if you use your Visa payWave card. Visa has a mainframe and uses it. Last I heard they've had a total of a very few seconds of outage (planned and unplanned combined) over the past decade plus — if you merely swiped (or waved) twice, you wouldn't have noticed.

UPDATE #3: To give you a better idea how serious this problem is, the Starbucks Singapore shop near my office has a corporate-issued (and professionally made) "Our cards aren't working" sign posted atop the counter. In other words, Starbucks cards are so unreliable the corporate office had to issue signs to its shops, similar in quality and appearance to its menu boards.

My dear Starbucks friends: do you make your coffee with portable electric tea kettles? Pick the right tool for the job. That would be a mainframe for payments. Mainframes work, and you can buy or rent one. In the meantime, if you need some help you can find me over at The Coffee Bean.

About "(Blank) Needs a Mainframe"

By now Mainframe Blog readers have seen several "...needs a mainframe" posts. We try to set some trends here, and that's the whole point about these posts.

The central premise (if you'll pardon the pun) of mainframe computing is about quality. Sure, you can add, subtract, multiply, divide, and branch on a PC or an iPod. Lots of computers are Turing-complete, including mainframes. But if you have a business or government to run, and if at least some of your business processes are important, then, quite simply, you need a mainframe — and you need to use it. Otherwise, it's going to be much harder to deliver the security, reliability, and other qualities real people increasingly demand.

The information technology industry solved these quality problems a long time ago, and the solutions to those problems involved relying on the highest quality infrastructure (i.e. mainframes) combined with centrally focused, highly disciplined operations and change management (i.e. mainframe-related development and operations), end-to-end. We know that formula works. Yet there are way too many businesses, governments, and their IT organizations that have lost the plot, implementing obscenely complex, Rube Goldberg-esque application architectures to fulfill even the most common and critical business functions. Such architectures are costly, fragile, and vulnerable.

Unfortunately, as we've seen over just the past few weeks, quality is deteriorating. Major businesses are crashing and burning, hard, with security and availability crises causing major disruptions. Public "cloud computing" isn't going very far unless quality improves dramatically and quickly. Only the fit will survive: the organizations that have or adopt mainframes and actually use them for their critical business processes, end-to-end. It's really that simple: "Fit for Quality."

One technology company that distinguishes itself on quality is the world's largest technology company: Apple. Here's a 30 second video example from 1995:

Apple is a remarkable company. Apple has mastered the "it just works" segment of the consumer technology market. As technology (and life) gets ever more complicated, and as the value of time increases, more and more people value technology like Apple's. The same is true in the world's data centers. Businesses and governments want solutions that deliver secure, reliable service. Those qualities are becoming more important every day. And I think IBM ought to press home its advantages and repeat this simple phrase:

U.S. Airways (Now United Airlines) Needs a Mainframe

U.S. Airways, which is merging with United Airlines, had to stop flying on Friday when key applications, notably their boarding system, became unavailable. The airline claims that a power outage near their (only?) data center in Phoenix caused the outage, which in turn caused chaos at U.S. Airways counters and boarding gates despite clear skies and good weather.

If U.S. Airways had a pair of IBM mainframes — one in their primary data center, one in a second data center — if they configured them in a remote cluster (using an appropriate flavor of Geographically Dispersed Parallel Sysplex), and if they actually used those mainframes to support their most critical business processes, end-to-end, then it's extremely unlikely they would have had this problem — and certainly not for hours. That particular infrastructure formula should be familiar. Was U.S. Airways following that formula? If not, why not?

EMC/RSA "SecurID" Compromised, Lockheed Martin Hacked

Have you seen those key fobs that display a new pseudo-random series of numeric digits every minute or so? To log onto a network or system you have to enter the current set of digits plus your regular credentials (user ID and password), typically.

Unfortunately a group of unknown hackers, possibly a group sponsored by a government, broke into EMC's RSA division and figured out how to duplicate those key fobs, in effect. Then the same group (perhaps) broke into Lockheed Martin, the leading U.S. defense contractor.

It's not clear what sensitive information was taken, and Lockheed Martin isn't saying. However, it's possible the invaders were able to find details about future weapons systems along with operational information about current military deployments in Afghanistan and Iraq, among other places.

I might have more to say in a subsequent post about mainframes, mainframe security systems, and their important role in "defense in depth" — a role which some businesses and governments are not exploiting to full advantage.