Author: Mike Bursell

I want to talk about security in DevOps – also known as “DevSecOps”* – on this blog, but it’s going to have to wait for a subsequest post. I’ve been speaking at a few conferences and seminars recently on this topic, and I’ve started to preface the discussion with a quick question: “Who here understands what a container is?”. And you don’t get that many hands going up**. So I’ve started giving a little explanation of what containers*** are before going much further.

To be clear: can you do DevOps without containers? Can you do DevSecOps without containers? The answer to both is “yes”. But containers lend themselves so well to the DevOps approach – and to DevSecOps, it turns out – that even though it’s possible to do DevOps without them, I’m going to assume that most people will use them.

What is a container?

I was in a meeting with a number of colleagues a few months ago, and one of them was presenting on containers. Not everybody around the table was particularly expert in the technology, so he was starting simple. He made a statement, which said something like “There’s no mention of containers in the Linux kernel source code”. This, it turned out, was a dangerous statement to make, given some of the attendees, and within a few seconds, both my boss (sitting next to me) and I were downloading the recent kernel source tarballs and performing a count of the exact number of times that the word “container” occurred. It turned out that his statement wasn’t entirely correct. To give you an idea, I’ve just tried it on an old version that I happened to have on my laptop (4.9.2), and I got a result of 15,273 lines including the word “container”****. My boss and I had a bit of a smirk and ensured that we corrected him at the next break.

What my colleague meant to say – and he clarified later – is that the concept of a container doesn’t really exist as a clear element within the Linux kernel. In other words, containers use a number of abstractions, components, tools and mechanisms from the Linux kernel, but there’s nothing very special about these: they can be used for other purposes as well. So, there’s “no such thing as a container, according to the Linux kernel”.

What, then, is a container? Well, I come from a virtualisation – hypervisor and VM – background, and the way that I tend to think of it is that containers are both very much like and very much unlike VMs. I realise that this may initially not sound very helpful, but let me explain.

How is a container like a VM?

The main way in which a container is like a container is like a VM is that it’s a unit of execution. You bundle something up – an image – which you can then run on a suitably equipped host platform. Like a VM, it’s a workload on a host, and like a VM, it runs at the mercy of that host. Beyond providing workloads with the resources that they need to do their job (CPU cycles, networking, storage access, etc.), the host has a couple of jobs that it needs to do:

protect workloads from each other, and make sure that a malicious, compromised or poorly written workload cannot affect the operation of any others;

protect itself (the host) from workloads, and make sure that a malicious, compromised or poorly written workload cannot affect the operation of the host.

The ways that this is achieved for VMs and containers is fundamentally different, with isolation, with VMs being kept isolated by hypervisors making use of hardware capabilities, and containers being kept isolated via software controls provided by the Linux kernel******. These controls revolve around various “namespaces” which ensure that one container can’t see other containers’ files, users, network connections, etc. – nor those of the host. These can be supplemented by tools such as SELinux, which provide capabilities controls for further isolation of containers.

How is a container unlike a VM?

The problem with the description above is that if you’re even vaguely hypervisor-aware, you probably think that a container is just like a VM, and it really isn’t.

A container is, first and foremost *******, a packaging format. “WHAT?” you’re saying, “but you just said it was something that was executed.” Well, yes, but much of the reasons containers are so interesting is that it’s very easy to create the images from which they’re instantiated, and those images are typically much, much smaller than for VMs. For this reason, they take up very little memory, and can be spun up and spun down very, very quickly. Having a container that sits around for just a few minutes, or even seconds (OK, milliseconds, if you like) is an entirely sensible and feasible idea. For VMs, not so much.

Given that containers are so lightweight and easy to replace, people have started using them to create micro-services – minimal components split out of an application which can be used by one or many other micro-services to built whatever you want. Given that you only plan to put what you need for a particular function or service within a container, you’re now free to make it very small, which means that writing new ones and throwing away the old ones becomes very practicable. We’ll follow up on this when we talk about DevOps next time. We’ll also talk about some of the impacts this might have on security, and hence DevSecOps.

Hopefully this has been a useful intro, and you feel excited and motivated to learn more about DevSecOps next time. (And if you don’t, just pretend.)

*I think because SecDevOps reads oddly, and DevOpsSec tends to get pluralised, and then you’re on an entirely different topic.

**I should note that this isn’t just to British audiences, who are reserved and don’t like drawing attention to themselves: this is to Canadian and US audiences who, well … are different in that regard.

***I’m going to be talking about Linux containers: I’m aware there’s history here, so it’s worth noting. In case of pedantry.

***** to be fair, a number of those uses, at a quick glance, have nothing to do with containers in the way we’re discussing them “Linux containers”, but refer to abstractions which can be said to container other elements, and are, therefore, logically referred to as containers.

****** there are clever ways to combine VMs and containers to benefit from the strengths of each – I’m not going to go into those today.

******* well, apart from the execution bit that we just covered, obviously.

History repeats itself because no one was listening the first time. (Anonymous)

We’re all fallible. You’re fallible, he’s fallible, she’s fallible, I’m fallible*. We all get things wrong from time to time, and the generally accepted “modern” management approach is that it’s OK to fail – “fail early, fail often” – as long as you learn from your mistakes. In fact, there’s a growing view that if you’d don’t fail, you can’t learn – or that your learning will be slower, and restricted.

The problem with some fields – and IT security is one of them – is that failing can be a very bad thing, with lots of very unexpected consequences. This is particularly true for operational security, but the same can be the case for application, infrastructure or feature security. In fact, one of the few expected consequences is that call to visit your boss once things are over, so that you can find out how many days*** you still have left with your organisation. But if we are to be able to make mistakes**** and learn from them, we need to find ways to allow failure to happen without catastrophic consequences to our organisations (and our careers).

The first thing to be aware of is that we can learn from other people’s mistakes. There’s a famous aphorism, supposedly first said by George Santayana and often translated as “Those who cannot learn from history are doomed to repeat it.” I quite like the alternative: “History repeats itself because no one was listening the first time.” So, let’s listen, and let’s consider how to learn from other people’s mistakes (and our own). The classic way of thinking about this is by following “best practices”, but I have a couple of problems with this phrase. The first is that very rarely can you be certain that the context in which you’re operating is exactly the same as that of those who framed these practices. The other – possibly more important – is that “best” suggests the summit of possibilities: you can’t do better than best. But we all know that many practices can indeed be improved on. For that reason, I rather like the alternative, much-used at Intel Corporation, which is “BKMs”: Best Known Methods. This suggests that there may well be better approaches waiting to be discovered. It also talks about methods, which suggests to me more conscious activities than practices, which may become unconscious or uncritical followings of others.

What other opportunities are open to us to fail? Well, to return to a theme which is dear to my heart, we can – and must – discuss with those within our organisations who run the business what levels of risk are appropriate, and explain that we know that mistakes can occur, so how can we mitigate against them and work around them? And there’s the word “mitigate” – another approach is to consider managed degradation as one way to protect our organisations***** from the full impact of failure.

Another is to embrace methodologies which have failure as a key part of their philosophy. The most obvious is Agile Programming, which can be extended to other disciplines, and, when combined with DevOps, allows not only for fast failure but fast correction of failures. I plan to discuss DevOps – and DevSecOps, the practice of rolling security into DevOps – in more detail in a future post.

One last approach that springs to mind, and which should always be part of our arsenal, is defence in depth. We should be assured that if one element of a system fails, that’s not the end of the whole kit and caboodle******. That only works if we’ve thought about single points of failure, of course.

The approaches above are all well and good, but I’m not entirely convinced that any one of them – or a combination of them – gives us a complete enough picture that we can fully embrace “fail fast, fail often”. There are other pieces, too, including testing, monitoring, and organisational cultural change – an important and often overlooked element – that need to be considered, but it feels to me that we have some way to go, still. I’d be very interested to hear your thoughts and comments.

*my family is very clear on this point**.

**I’m trying to keep it from my manager.

***or if you’re very unlucky, minutes.

****amusingly, I first typed this word as “misteaks”. You’ve got to love those Freudian slips.

Blockchains are big news at the moment. There are conferences, start-ups, exhibitions, open source projects (in fact, pretty much all of the blockchain stuff going on out there is open source – look at Ethereum, zcash and bitcoin as examples) – all we need now are hipster-run blockchain-themed cafés*. If you’re looking for an initial overview, you could do worse than the Wikipedia entry – that’s not the aim of this post.

Before we go much further, one useful thing to know about many blockchains projects is that they aren’t. Blockchains, that is. They are, more accurately, distributed ledgers****. For now, however, let’s roll in blockchain and distributed ledger technologies and assume we’re talking about the same thing: it’ll make it easier for now, and in most cases, the difference is immaterial for our discussions.

I’m not planning to go into the basics here, but we should briefly talk about the main link with crypto and blockchains, and that’s the blocks themselves. In order to build a block, a set of transactions to put into a blockchain, and then to link it into the blockchain, cryptographic hashes are used. This is the most obvious relationship that the various blockchains have with cryptography.

There’s another, equally important one, however, which is about identity*****. Now, for many blockchain-based crypto-currencies, a major part of the point of using them at all is that identity isn’t, at one level, important. There are many actors in a crypto-currency who may be passing each other vanishingly small or eye-wateringly big amounts of money, and they don’t need to know who each other is in order to make transactions. To be more clear, the uniqueness of each actor absolutely is important – I want to be sure that I’m sending money to the entity who has just rendered me a service – but being able to tie that unique identity to a particular person IRL****** is not required. To use the technical term, such a system is pseudonymous. Now, if pseudonymity is a key part of the system, then protecting that property is likely to be important to its users. Crypto-currencies do this with various degrees of success. The lesson here is that you should do some serious reading and research if you’re planning to use a crypto-currency, and this property matters to you.

On the other hand, there are many blockchain/distributed ledger technologies where pseudonymity is not a required property, and may actually be unwanted. These are the types of system in which I am most generally interested from a professional point of view.

In particular, I’m interested in permissioned blockchains. Permissionless (or non-permissioned) blockchains are those where you don’t need permission from anyone in order to participate. You can see why pseudonimity and permissionless blockchains can fit well today: most (all?) crypto-currencies are permissionless. Permissioned blockchains are a different kettle of fish, however, and they’re the ones at which many businesses are looking at the moment. In these cases, you know the people or entities who are going to be participating – or, if you don’t know now, you’ll want to check on them and their identity before they join your blockchain (or distributed ledger). And here’s why blockchains are interesting in business********. It’s not just that identity is interesting, though it is, because how you marry a particular entity to an identity and make sure that this binding is not spoofable over the lifetime of the system is difficult, difficult, lemon difficult******** – but there’s more to it than that.

What’s really interesting is that if you’re thinking about moving to a permissioned blockchain or distributed ledger with permissioned actors, then you’re going to have to spend some time thinking about trust. You’re unlikely to be using a proof-of-work system for making blocks – there’s little point in a permissioned system – so who decides what comprises as “valid” block, that the rest of the system should agree on? Well, you can rotate around some (or all) of the entities, or you can have a random choice, or you can elect a small number of über-trusted entities. Combinations of these schemes may also work. If these entities all exist within one trust domain, which you control, then fine, but what if they’re distributors, or customers, or partners, or other banks, or manufacturers, or semi-autonomous drones, or vehicles in a commercial fleet? You really need to ensure that the trust relationships that you’re encoding into your implementation/deployment truly reflect the legal and IRL trust relationships that you have with the entities which are being represented in your system.

And the problem is that once you’ve deployed that system, it’s likely to be very difficult to backtrack, adjust or reset the trust relationships that you’ve designed in. And if you don’t think about the questions I noted above about long-term bindings of identity, you’re going to be in some serious problems when, for instance:

an entity is spoofed;

an entity goes bankrupt;

an entity is acquired by another entity (buy-outs, acquisitions, mergers, etc.);

an entity moves into a different jurisdiction;

legislation or regulation changes.

These are all issues that are well catered for within existing legal frameworks (with the possible exception of the first), but which are more difficult to manage within the sorts of systems with which we are generally concerned in this blog.

Please don’t confuse the issues noted above with the questions around how to map legal agreements to the so-called “smart contracts” in blockchain/distributed ledger systems. That’s another thorny (and, to be honest, not unconnected issue), but this one goes right to the heart of what a system is, and it’s the reason that people need to think very hard about what they’re really trying to achieve when they adopt our latest buzz-word technology. Yet again, we need to understand how systems and the business work together, and be honest about the fit.

*if you come across one of these, please let me know. Put a picture in a comment or something.**

**even better – start one yourself. Make sure I get an invitation to the opening***.

***and free everything.

****there have been onlines spats about this. I’m not joining in.

*****there are others, but I’ll save those for another day.

******IRL == “In Real Life”. I’m so old-skool.

*******for me. If you’ve got this far into the article, I’m hoping there’s an evens chance that the same will go for you, too.

********I’ll leave this as an exercise for the reader. Watch it, though, and the TV series on which it’s based. Unless you don’t like swearing, in which case don’t watch either.

Two weeks ago, I didn’t write a full post, because the Manchester arena bombing was too raw. We are only a few days on from the London Bridge attack, and I could make the same decision, but think it’s time to recognise that we have a new reality that we need to face in Britain: that we may live in a new security and terrorism landscape. The sorts of attacks – atrocities – that have been perpetrated over the past few weeks (and the police and security services say that despite three succeeding, they’ve foiled another five) are likely to keep happening.

And they’re difficult to predict, which means that they’re difficult to stop. There are already renewed calls for tech companies* to provide tools to allow the Good Guys[tm**] to read the correspondence of the people who are going to commit terrorist acts. The problem is that the preferred approach requested/demanded by governments seems to be backdoors in encryption and/or communications software, which just doesn’t work – see my post The Backdoor Fallacy – explaining it slowly for governments. I understand that “reasonable people” believe that this is a solution, but it really isn’t, for all sorts of reasons, most of which aren’t really that technical at all.

So what can we do? Three things spring to mind, and before I go into them, I’d like to make something clear, and it’s that I have a huge amount of respect for the men and women who make up our security services and intelligence community. All those who I’ve met have a strong desire to perform their job to the best of their ability, and to help protect us from people and threats which could damage us, our property, and our way of life. Many of these people and threats we know nothing about, and neither do we need to. The job that the people in the security services do is vital, and I really don’t see any conspiracy to harm us or take huge amounts of power because it’s there for the taking. I’m all for helping them, but not at the expense of the rights and freedoms that we hold dear. So back to the question of what we can do. And by “we” I mean the nebulous Security Community****. Please treat these people with respect, and be aware they they work very, very hard, and often in difficult and stressful jobs*****.

The first is to be more aware of our environment. We’re encouraged to do this in our daily lives (“Report unaccompanied luggage”…), but what more could we do in our professional lives? Or what could we do in our daily lives by applying our professional capabilities and expertise to everyday activities? What suspicious activities – from traffic on networks from unexpected place to new malware – might be a precursor to something else? I’m not saying that we’re likely to spot the next terrorism attack – though we might – but helping to combat other crime more effectively both reduces the attack surface for terrorists and increases the available resourcing for counter-intelligence.

Second: there are, I’m sure, many techniques that are available to the intelligence community that we don’t know about. But there is a great deal of innovation within enterprise, health and telco (to choose three sectors that I happen to know quite well******) that could well benefit our security services. Maybe your new network analysis tool, intrusion detector, data aggregator has some clever smarts in it, or creates information which might be of interest to the security community. I think we need to be more open to the idea of sharing these projects, products and skills – proactively.

The third is information sharing. I work for Red Hat, an Open Source company which also tries to foster open thinking and open management styles. We’re used to sharing, and industry, in general, is getting better about sharing information with other organisations, government and the security services. We need to get better at sharing both active data from systems which are running as designed and bad data from systems that are failing, under attack or compromised. Open, I firmly believe, should be our default state*******.

If we get better at sharing information and expertise which can help the intelligence services in ways which don’t impinge negatively on our existing freedoms, maybe we can reduce the calls for laws that will do so. And maybe we can help stop more injuries, maimings and deaths. Stand tall, stand proud. We will win.

*who isn’t a tech company, these days, though? If you sell home-made birthday cards on Etsy, or send invoices via email, are you a tech company? Who knows.

**this an ironic tm***

***not that I don’t think that there are good guys – and gals – but just that it’s difficult to define them. Read on: you’ll see.

****I’ve talked about this before – some day I’ll define it.

*****and most likely for less money than most of the rest of us.

******feel free to add or substitute your own.

*******OK, DROP for firewall and authorisation rules, but you get my point.

Let’s start with a story. Way back in the mists of time*, I performed audits for an organisation which sent out cryptographic keys to its members. These member audits involved checking multiple processes and systems, but the core one was this: the keys that were sent out were are really big deal, as they were the basis from which tens of thousands of other keys would be derived. So, the main key that was sent out was really, really important, because if it got leaked, the person who got hold of it would have a chance to do many, many Bad Things[tm].

The main organisation thought that allowing people the possibility to do Bad Things[tm] wasn’t generally a good idea, so they had a rule. You had to follow a procedure, which was this: they would send out this key in two separate parts, to be stored in two different physical safes, to be combined by two different people, reporting to two different managers, in a process split into to separate parts, which ensured that the two different key-holders could never see the other half of the key. The two separate parts were sent out by separate couriers, so that nobody outside the main organisation, could ever get to see the two parts. It was a good, and carefully thought out process.

So one of the first things I’d investigate, on arriving at a member company to perform an audit, would be how they managed their part of this process. And, because they were generally fairly clued up, or wouldn’t have been allowed to have the keys in the first place, they’d explain how careful they were with the key components, and who reported to whom, and where the safes were, and back up plans for when the key holders were ill: all good stuff. And then I’d ask: “And what happens when a courier arrives with the key component?” To which they’d reply: “Oh, the mail room accepts the package.” And then I’d ask “And when the second courier arrives with the second key component?” And nine times out of ten, they’d answer: “Oh, the mail room accepts that package, too.” And then we’d have a big chat.**

This is a classic example of a single point of failure. Nobody designs systems with a single point of failure on purpose****, but they just creep in. I’m using the word systems here in the same way I used it in my post Systems security – why it matters: in the sense of a bunch of different things working together, some of which are likely to be human, some of which are likely to be machine. And it’s hard to work out where single points of failure are. A good way to avoid them – or minimise their likelihood of occurrence – is to layer or overlap systems*****. What is terrible is when two single points of failure are triggered at once, because they overlap. From the little information available to us, this seems to be what happened to British Airways over the past weekend: they had a power failure, and then their backups didn’t work. In other words, they had a cascade failure – one thing went wrong, and then, when another thing went wrong as well, everything fell over. This is terrible, and every IT professional out there ought be cringing a little bit inside at the thought that it might happen to them.******

How can you stop this happening? It’s hard, really it is, because the really catastrophic failures only happen rarely – pretty much by definition. Here are some thoughts, though:

look at pinch points, where a single part of the system, human or machine, is doing heavy lifting – what happens when they fail?

look at complex processes with many interlocking pieces – what happens if one of them produces unexpected results (or none)?

look at processes with many actors – what happens if one or actor fails to do what is expected?

look at processes with a time element to them – what will happen if an element doesn’t produce results when expected?

try back-tracking, rather than forward-tracking. We tend to think forwards, from input to output: try the opposite, and see what the key parts to any output are. This may give unexpected realisations about critical inputs and associated components.

Last: don’t assume that your systems are safe. Examine, monitor, test, remediate. You might******* also have a good long hard think about managed degradation: it’s really going to help if things do go horribly wrong.

Oh – and good luck.

*around ten years ago. It feels like a long time, anyway.

**because, in case you missed it, that meant that the person in charge of the mail room had access to both parts of the key.***

***which meant that they needed to change their policies, quickly, unless they wanted to fail the audit.

****I’m assuming that we’re all Good Guys and Gals[tm], right, and not the baddies?

*****the principle of defence in depth derives from this idea, though it’s only one way to do it.

******and we/you shouldn’t be basking in the schadenfreude. No, indeed.

I’m not going to write a post today. Please remember those who have lost people they love, the injured and those who care for and about them. And if you struggle to be positive, remember the helpers: all those who came, and are coming, to their aid.

As you may have noticed*, there was somewhat of a commotion over the past week when the WannaCrypt ransomware infection spread across the world, infecting all manner of systems**, most notably, from my point of view, many NHS systems. This is relevant to me because I’m UK-based, and also because I volunteer for the local ambulance service as a CFR. And because I’m a security professional.

I’m not going to go into the whys and wherefores of the attack, of the importance of keeping systems up to date, the morality of those who spread ransomware***, how to fund IT security, or the politics of patch release. All of these issues have been dealt with very well elsewhere. Instead, I’m going to discuss talking to people.

I’m slightly hopeful that this most recent attack is going to have some positive side effects. Now, in computing, we’re generally against side effects, as they usually have negative unintended consequences, but on Monday, I got a call from my Dad. I’m aware that this is the second post in a row to mention my family, but it turns out that my Dad trusts me to help him with his computing needs. This is somewhat laughable, since he uses a Mac, which employs an OS of which I have almost no knowledge****, but I was pleased that he even called to ask a question about it. The question was “am I safe from this ransomware thing?” The answer, as he’d already pretty much worked out was, “yes”, and he was also able to explain that he was unsurprised, because he knew that Macs weren’t affected, and because he keeps it up to date, and because he keeps backups.

Somebody, somewhere (and it wasn’t me on this occasion) had done something right: they had explained, in terms that my father could understand, not only the impact of an attack, but also what to do to keep yourself safe (patching), what systems were most likely to be affected (not my Dad’s Mac), and what do to in mitigation (store backups). The message had come through the media, but the media, for a change, seemed to have got it correct.

I’ve talked before about the importance of informing our users, and allowing them to make choices. I think we need to be honest, as well, about when things aren’t going well, when we (singularly, or communally) have made a mistake. We need to help them to take steps to protect themselves, and when that fails, to help them clear things up.

And who was it that made the mistake? The NSA, for researching vulnerabilities, or for letting them leak? Whoever it was leaked them? Microsoft, for not providing patches? The sysadmins, for not patching? The suits, for not providing money for upgrades? The security group, putting sufficient controls in place to catch and contain the problem? The training organisation for not training the users enough? The users, for ignoring training and performing actions which allowed the attack to happen?

Probably all of the above. But, in most of those cases, talking about the problem, explaining what to do, and admitting when we make a mistake, is going to help improve things, not bring the whole world crashing down around us. Talking, in other words, to “real” people (not just ourselves and each other*****): getting out there and having discussions.

Sometimes a lubricant can help: tea, beer, biscuits******. Sometimes you’ll even find that “real” people are quite friendly. Talk to them. In words they understand. But remember that even the best of them will nod off after 45 minutes or so of our explaining our passion to them. They’re only human, after all.

*unless you live under a rock.

**well, Windows systems, anyway.

***(scum).

****this is entirely intentional: the less I know about their computing usage, the easier it is for me to avoid providing lengthy and painful (not to mention unpaid) support services to my close family.

*****and our machines. Let’s not pretend we don’t do that.

******probably not coffee: as a community, we almost certainly drink enough of that as it is.