Published

The Oncoming Train of Enterprise Container Deployments

As many of you know, adoption of containers has skyrocketed over the last year or two. Thus far, containers have been used mostly by early adopters, yet over the next several years we can expect widespread enterprise adoption. In fact, I expect the rate of adoption to exceed that of cloud (IaaS services), or virtualization before that. While it took enterprises perhaps a decade to fully plan and implement their virtualization initiatives, we can expect many enterprises to have production container deployments within three to five years. I fear that many of these implementations will have serious problems. Worse still, container technologies, when misused, inherently force us to own bad solutions for far longer.

Bear in mind that I am definitely not calling enterprises out specifically for misuse of container technology. Dismissive sentiments like “and that’s why we can’t have nice things” are not what I’m after. Enterprise adoption merely amplifies, by virtue of scale, the effects of anti-patterns in any technology, and containers are no exception. I am also not here to make fun of Docker for being on the Gartner hype cycle and being just short of the peak of inflated expectations. Container technology is sufficiently mature to be useful today and runtime environments like Kubernetes or Mesosphere are well on their way to becoming widely usable. In a couple years, they may even be classified as “boring technology” so that nobody, even the late majority or laggards, would feel it risky to use any of this technology.

Rather, this post is simply about the horrifying realization that containerization opens up a whole new playing field for folks to abuse. Many technology professionals in the coming decades will bear the brunt of the mistakes people are making today in their use of containers. Worse, these mistakes will be even more long-lived, because containers — being portable artifacts independent of the runtime — can conceivably survive in the wild far longer than, say, a web application written using JSP, Struts 1.1 and running under Tomcat 3.

Here are a few antipatterns I see out in the wild as container adoption spreads.

Using containers without really understanding why

The reason that “Docker Docker Docker” is such a meme is because as with any new technology, there are large swaths of unqualified technology “professionals” ready and eager to rub it all over everything. For those of you who have been around the block a few times, you probably saw this during the rise of virtualization. I once had enterprise architecture hand down an edict saying that we needed to virtualize the whole data center. (“Including the VAX 11/780 and all the Novell NetWare servers?” I asked.) In fact, they initially convinced the capital budgeting folks to only provide IT a fraction of the requested budget “because you’re going to virtualize everything at an 8:1 ratio like Gartner says, right?” I can only imagine there are equally clueless TOGAF-certified EAs running around today telling IT departments they must containerize everything, whether those workloads are appropriate for containerization or not. And EAs aren’t the only ones to blame. There’s no shortage of developers in startups insisting that Docker is the “future” and that it must be used for everything.

Blindly containerizing legacy applications

This brings me to the related anti-pattern of people who insist on jamming legacy applications into containers. This makes no sense. Containers are designed for per-process isolation, ergo, one process per container. Containers aren’t going to be particularly useful for running Oracle 12c or SAP HANA that have a million processes. You could containerize them, but why? If a service is not something you’ll be ripping and replacing frequently, there’s really no point.

I also see container technologies being used to keep legacy applications alive forever, which is even more terrifying. In the past, hardware refresh cycles or operating system EOL deadlines could serve as forcing functions to get rid of, say, that legacy PHP 3 application. Now, IT departments will have absolutely no incentive to do so when they can just “containerize the legacy”. We can therefore expect to still be running today’s applications, inside containers, in 2050 or beyond.

Ignoring devops principles

Back when I worked in operations, it was common for software developers to throw artifacts like JAR or WAR files over the wall for us to run. While terrible, at least the blast radius was reasonably predictable. Any senior application administrator could sandbox the application to constrain what it could do, and when things went wrong, there were standard ways of figuring out what the nature of the problem was.

In a containerized world, however, it gets way worse for teams that haven’t adopted devops practices. Now, these application teams aren’t just throwing Java bytecode over the wall; they’re throwing entire machine images. Think of all the things that can go wrong on a Linux system and how much maintenance needs to be done to keep it running smoothly, like security patching, rebooting, performance tuning, and so on. Now apply all that work to a microservices world, where you could have tens or hundreds of thousands of containers — which are really full-blown Linux systems — running in production. Keeping the lights on will become a nightmare without good cooperation and shared responsibility between dev and ops. Sadly, I see many container initiatives being driven primarily by application teams who have sold containers to the business as a value delivery accelerator, and then forcing operations teams to own it. This will come down crashing down on both groups when production failures occur.

Lack of standard quality assurance principles

One major gap in the container world is a standard workflow for verifying correctness of the images that one is building. In fact, since the premise is that any developer can build an image on his/her workstation and push it to a registry, or worse, right to a runtime, the workflow by definition doesn’t have any quality gates. How are enterprises to know that the image functions correctly? Or is compliant, or secure? Or that it will continue to be that way even if changes are made? Until container adopters realize that everything they learned about ensuring safety in the delivery of software products doesn’t get thrown away in a container universe, we can expect a lot of buggy, insecure applications being deployed to production or pushed to the Docker Hub where, like a virus, they will get used by others. (Oops, that’s already happened.)

Giant container images containing all of userland

By default, most containers provide a full Linux system, with all of /bin, /usr, and so on. Too few people build base images, or start container builds from scratch (literally FROM scratch in Docker). It’s too easy to write FROM ubuntu:14.04 and go from there. Now you’re responsible for all of Linux again even though you’re running one process in it.

Unfortunately, I believe that the “large attack surface” is a fundamental design problem with containers being an evolutionary, not a revolutionary step from VMs and bare metal. Container technology has been so successful purely in its ability to efficiently de-duplicate and package an entire Linux userland in a portable, run-anywhere image. But the application process still believes it is running inside a full multi-user Linux OS, with all of the kernel’s syscalls available to it, a full TCP/IP network stack, and so on. This makes no sense. Why should my aforementioned microservice need to run in an environment where the 1800 utilities in /bin are available to it?

There are compensating measures (like the from-scratch builds I mentioned above) but I believe that this fundamental design flaw will lead to unikernels disrupting containers. Unikernels truly are revolutionary by sharing little heritage with today’s multiuser Linux OS. Yet unikernels have the potential to provide exactly the same runtime efficiency benefits as containers, and better, by running individual processes as close to the hardware as possible. Unikernels are also restrictive enough that they will prevent users from making the same kinds of mistakes they are currently making with container adoption, though I’m sure users will find other horrible ways to abuse them.

Unikernels today lack the polish and great user experience of containers. In fact, they have a long way to go to be usable by nearly anyone other than academic researchers. Eventually, though, some entrepreneur will build their next PaaS based on emerging unikernel technology (i.e. build the dotCloud of unikernels) and they will catch fire. Until then, we’ll need to let the boom-bust cycle of container technology run its course.

Innovators invent a technology to solve a particular use case. In the case of Docker, dotCloud married LXC with a deployable artifact in order to run a PaaS and thus containers were born.

Early adopters evangelize and use it because it’s so much better than what is out there. The use cases are fairly congruent with the innovators.

The early majority comes along and half the time uses the technology well, and half the time uses it poorly. In either case, these folks extend the technology into other use cases, for better or worse.

The late majority copy both patterns and anti-patterns established by the early majority. The ratio of good to bad solutions stays roughly the same, but there are just more of each.

The laggards end up owning whatever solutions everyone else came up with, good or bad. 🙂 Everyone else in the adoption cycle has moved onto other things by then.

Technologies last a very long time. Mikey Dickerson of the U.S. Digital Service spoke recently about how Medicare still primarily runs on 7000 COBOL jobs written piecemeal over the last 30 years. Nobody has a complete map of the dependencies between these jobs, or where to even start replacing them. Because patterns are so commonly copied, it’s important that for future generations, we try to minimize the bad solutions and make more good ones.

Sadly, technology hype creates a reality distortion field that stops people from clearly assessing whether their use of that technology is a) appropriate and b) architected well. This leads me to believe that people will eventually abandon containerization and flock to something like unikernels. This won’t be because the technology is better, which it is, but simply because unikernels will be a new way of leaving behind poorly-architected container solutions, much as containers today are often seen as a way to leave behind all that “old legacy crap” that no forward-thinking developer wants to touch.

I disagree with “don’t package legacy apps”. The reason people do it is so you dont have to keep 5-10 years out of date distro running on machine just because your shitty vendor is out of date or your management doesnt want to buy upgrade.

Instead of keeping whole stack you can put just what app needs in container and keep rest of the system up to date

“I see many container initiatives being driven primarily by application teams who have sold containers to the business as a value delivery accelerator, and then forcing operations teams to own it.”

I think this is a key point. Devs are mandated with new product & delivering features faster. Ops on the other hand is tasked with stability & reliability. So they turn out to have very different perspectives & time horizons.

Let’s hope “devops” means devs & ops all work together on these initiatives. I think you’re right though when a new trend takes hold it make take a few accidents before people start to be more considered & cautious.