Puppet Podcast: Security with Ben Hughes of Etsy

Breadcrumb

The Puppet Podcast is back with a three-part series on security. “But wait, isn’t Puppet all about DevOps?” you ask. Here’s the thing: People have always used Puppet for security. (Check out Bill Weiss’ talk at PuppetConf to learn more.) In this episode, we chat with Ben Hughes from Etsy about DevOps and security working together. Turns out, security professionals are people too, and having empathy for them can go a long way to improving relations with them.

In the second half, we talk gender diversity. As many people know, InfoSec consists of 10 percent women and underrepresented groups. I think we can do better, and doing so will make the industry, as a whole, better. Diversity in thought, approach, and problem solving is needed everywhere.

We also chat about various industry issues, including industry shifts and where Puppet is growing its security and compliance offering. And there may be a wisecrack or two about fedoras.

A transcript of the podcast is below.

If you like what you’ve heard, check out Ben Hughes’ blog post on Imposter Syndrome. It’s really well done, and I suspect will resonate whether you work in security or not. I also recommend two DevOpsDays PDX talks:

TRANSCRIPT

Kara Sowles, Puppet community manager: Hi, everyone. Welcome to the Puppet podcast. I'm [Kara]. I'm your usual host. But today, I'm so excited to be introducing our security podcast miniseries. Beth from Puppet is here and will be leading the series of podcasts on — you guessed it — security.

We've still got plenty of Puppet-filled podcasts ready for you as well. Until then, Beth, do you want to talk a little bit about why we find this topic so important as to dedicate a whole miniseries to it?

Beth Cornils, senior product manager, Puppet: Yes, Kara. I would love to talk about that. In the world today, security is really important. It's important for a variety of reasons. One of the things that I love about the first podcast that we're doing to kickoff is that a lot of people don't understand that every single person within the organization plays a part within security.

It doesn't matter if you're the receptionist or if you're the security architect. Everybody is important. So the goal is to talk about security, to make the security people real people, to help everyone understand why it's important.

Kara: Excellent. That's — actually, I'm hearing a lot of pressure for me even as the community manager it's — I'm part of security too.

Beth: You are turns out —

Kara: Uh-oh.

Beth: — a really important one. [laughter]

Kara: Well, who is our first guest here today?

Beth: So our first guest is Ben Hughes, the delightful Ben Hughes.

Ben Hughes, security monkey at Etsy: Hello.

Beth: So — hello. And while I've never personally met Ben, I have heard great things about him. So I've also been sent to a number of his videos and podcasts. And I would recommend everybody else do the same thing. I find them entertaining and informative.

You can't find that very often in a video, especially about security. I mean, come on. And also, he did this amazing write-up about imposter syndrome in security, which I think is relevant for everyone, not just people in security.

So that was — I would encourage everyone to kind of go down that path and read it if you haven't already. So I kind of feel like I've talked enough at this point. So let's turn it over to Ben. Ben, tell us a little about you.

Ben Hughes, security expert, Etsy: Hi. I'm Ben. I work for Etsy. I'm based in San Francisco [at Etsy, predominantly] based on New York, Brooklyn, but they have offices throughout the world. Before that, I was actually at Puppet Labs, as it was back then.

Kara: I thought I met you somewhere before.

Ben Hughes: [Reductive Labs community leader]. [laughter] I went to the first Puppet camp back in like the 1800s, which was the first-round interview for everyone at Puppet now. So I've been involved in the Puppet world — was involved in the Puppet community and seemingly still am, which is why I'm here. Before that, I've — in a number of successful and failed startups throughout London and other places in the world.

Beth: Excellent. So one of the things that I wanted to talk about is that people often assume that, when they're going to talk to a security person, that the first thing the security person is going to tell them is no. No, you can't do that, whether you're a developer, whether you're ops. It doesn't matter who you are. They're going to go to security.

And it's like you're going to security because you're in trouble. So I wanted to get your perception about, over the years, kind of how you've handled that and if you've seen that that's changed at all and if it depends on the organization or maybe like how you've changed over time.

Ben Hughes: Yeah. I've definitely changed over time. I remember, 15 years ago in one of my first security roles, I was quite young, pretty immature, not always the most thoughtful about how my actions would impact anyone else. I've certainly taken out a lot of Solaris servers by trying to find vulnerabilities in them. But that's kind of your own fault.

I think the industry or at least parts of the industry has matured over the years. Certainly, I'd love to hope I have. But the jury is still out. If you just say no to people, then they'll ignore you and go around you rather than actually doing the right thing.

So to steal the line from the exciting world of improv, saying. "Yes. And what can I do instead?" or, "Can we compromise?" or, "Do you really need this access?" or, "How can we actually redesign this in a sensible way?" and actually thanking them for coming and talking to you rather than punishing them because not that many people like just being punished for asking a fairly innocuous question.

Beth: I would imagine — and you can correct me if I'm wrong — that there's a difference between a developer and working with a developer in terms of from a security perspective and sort of the no aspect and someone working in HR or finance or someone who maybe is not in developing the code that goes into the features that get sent out.

Ben Hughes: Yeah. I think security actually needs to look at what the actual impact is and what people are trying to do. The majority of people in finance or developers or any part of your organization generally don't wake up in the morning and go, "How can I be the most malicious and unhelpful to security I can?"

They, like everyone else, just want to get their job done so that they can go home and [endure] Netflix. So how you like help them do that in a way that doesn't alienate them from you is actually the job of security not to just like block everyone from doing everything.

Because it turns out blocking everyone from doing everything is not a great way to make money. And yeah capitalism.

It's really not it turns out. No. As I've been working on some of the security stuff for Puppet, that's one of the things that I've heard from people all over. It's like the most secure thing is to just unplug everything and don't let anybody do work. It turns out, can't do that.

Ben Hughes: Yeah. It's not — I don't know that many successful businesses that have taken all their computers and filled them with concrete.

Beth: [laughs] I don't either. It's a [very weird thing].

Ben Hughes: But I didn't study economics. So I'm not an MBA. So I could be way off base there.

Beth: [laughs] You could. But I think you're probably right.

Ben Hughes: Yeah. The phrase security nihilism, which Alex Stamos of formerly Yahoo and now Facebook has kind of coined, rubs into that view of it being very much [idio-secure], as in it's smashed into tiny pieces and then launched into the sun, or it's not secure. And it turns out there is a step in between those, in fact numerous steps in terms of how much time, effort and money you're willing to throw at securing things and how important things are.

Beth: So with that, especially from a developer's perspective, how — or ops — how do you get people comfortable with the fact that nothing is ever going to be 100 percent secure? So like you saying, "No, but," or "Hey, have you thought about the consequences of this?"

How do you approach that so that they understand that you understand that nothing is ever going to be 100 percent secure? I'm assuming you understand that. Should I have clarified that first?

Ben Hughes: So I think developers and operations people certainly understand that better than security people. Security people — this trend is beginning to erode. But certainly — and I've been as guilty of this and brilliantly continue to be of like, "We can't do this. There are ways around it. Or there are bypasses. Or there are flaws in it."

And you're like, sure. But like most homes have terrible locks on them. It turns out that's fine because the majority of criminals don't pick locks. That's actually quite rare. They do have things like bricks or wait till people leave their door open.

And like spending like thousands of dollars on an amazing set of locks for like a $10 wooden door frame, also pointless. So they kind of get lost in those weeds.

Whereas, if you ask a developer, "Does your code have any bugs then?" all but the most arrogant will go, "Well, yeah. Of course." And we'll find those. And that will be exciting. And you ask an ops person, "So we need 100 percent up time on this."

And anyone with their beans will go, "You don't pay me enough to make that 100 percent uptime. And we're going to need a much bigger budget to do that," because those are like untenable goals.

But you say to a security person, "We need to make this 100 percent secure," and you'll either have some that are still laughing to this day about that notion or some that go, "Okay. Then, we need to do all these things." And it's not going to happen.

Beth: Right. Okay. So how do you bring awareness to security within an organization without making people completely paranoid? Because I know, once I started investigating and getting into, as a product manager into the world of security, all I could think is, my gawd, the world is burning.

And it seemed a little bit terrifying. So are there ways to make it fun so that people understand that, hey, I do play a part, and it's not the end of the world if something gets screwed up?

Ben Hughes: Yeah. I'd say two parts to that. So with the whole the world is burning, it is. But the world has also been going quite a while. And it hasn't magically ended yet. So as insecure as you were yesterday, just because you have new knowledge about that, that doesn't actually change anything. It just makes you paranoid but not necessarily with too much more real reason.

Beth: Well, not everybody is as Zen as you are or have been doing this as long as you are.

Ben Hughes: I'm very glad people aren't as Zen as me, as there would have to be a lot more sleeping tablets. On the raising awareness, certainly at Etsy, we have a couple of concepts for doing that.

One of them is security candy where we just spend lots of money on candy and have jars of it at our desk, which encourages people to swing by our desks and, inadvertently, sometimes have to talk to us and build up rapport with us and occasionally even talk to us about security things, all while slowly killing themselves through fructose.

And I think we have an unsaid policy of all emails from us by near law must have some amusing GIF on them so that you know it's from the security team because there's — if you're going to give people bad news, at least give them like a dog falling over or something equally wild to lighten the load, which also like occasionally doubles as a, "Was this really sent by you? There was no ridiculous animated animal associated with it."

We're like, "No. That was. We just missed it." So I think those seemingly small things actually do quite a lot. And I know my friend [Astira] at SoundCloud actually prints — you call them buttons. The free world calls them badges with like a button press, badge press. I'm looking at Kara —

Kara: Yeah. It's a button.

Ben Hughes: [Because you most certainly] have like six of these [at your house].

Kara: Yeah. No. I have a whole box of buttons but no button maker. So if you've got a button maker, we should talk later.

Ben Hughes: I know. You can probably buy one off Etsy.com. [laughter] It makes those for when people do good security things. If like you reported some phishing, you get a button.

That actually — their ridiculous gamification of buttons — they're still badges — seems to work quite well, fun things like that. Security doesn't have to be all tears and sadness.

Beth: It doesn't? Gawd, that's good to know because, wow, it's been kind of depressing.

Ben Hughes: It's still depressing. But you're less depressed with a button.

Beth: Okay. Okay. You're right. That can make the whole world better.

Ben Hughes: Yeah.

Beth: The whole world gets better with a button.

Ben Hughes: Certainly better than most things vended than it is of RSA or blackout.

Beth: Yeah. Yeah.

Ben Hughes: So —

Beth: Okay. I'll give you that. Fair enough. And I guess we've kind of already talked about that because, you know, I prepared, and we didn't really go off my script kind of like what you said. That was awesome. Thanks for the heads up.

Beth: [laughs] What? So what do you wish people understood about your area of security and what you do? And do you think that that would help understanding with kind of — so here's what I'm kind of looking for is the humanization —

Ben Hughes: What do you want me to say?

Beth: I want you to answer from your heart. And — [laughs] so let me give you an example. Oftentimes, I go into meetings. And people will say, "Well, product didn't tell us. So we didn't do it."

And it's like, hi. My name is Beth. I am product. I'm right here. So can you say, "Beth, I need some information from you"? Does security have that same issue that it's like you're security; you're not Ben who works in security?

And what would you want people to understand about your day-to-day life that kind of like humanizes you? Because we all have bad days. We all have good days. Sometimes, we like — I can't swear — screw things up. There's all kinds of things that can go wrong in a given day.

Sometimes, you can have an entire bad week. And it doesn't make you a bad — I know. But it doesn't make you a bad person. So how do you kind of like —

Ben Hughes: In security, we dream of only having bad weeks as opposed to like bad epochs. I blame Solaris. Humanizing us — I d — I visit a lot of security people out there who probably aren't that human and who are very good at like smashing out bugs or reversing things or writing exploits.

But those skills are amazing and years beyond me. I'm never going to be that smart. But they're not the area I specialize in. And they're not what I spend my time doing. I have always — that's not true.

Since being in the world of employment, have always been trying to defend things, which is a different skillset in that you have to know what you have. And you have to know what's valuable. And you have to defend it that way.

So I've not suffered too much from just being an amorphous blob of security. I'm sorry, in product, that you are.

Kara: This is a pretty familiar topic, I think, for a lot of — I know we have a lot of listeners that are sys administrators and work in IT. Like this question kind of comes up a lot I think in those contexts as well.

Beth: Right.

Ben Hughes: I think, to associate it back to that, the struggle I've had in days doing systems work is, if — and this isn't to start the sales and engineering war that has been hashed to death in the world.

But if sales has a good week, everyone is like cheers. Yay. You've sold this much stuff. If the website still works, no one goes around high-fiving your sys admins. They're like, yeah. Still works. Lights are still on.

Security is like that. And to have my own bleeding-heart story — but even worse because I'm very hard done by — that like we don't think we got hacked this week because it's really hard to prove that. Yay. It's like an even less empty, hollow victory than the website is still up.

It's like as far as we're aware with the tools we have available, we haven't been owned by a large nation state who are going to do something in like two years' time, which is not the same as dehumanizing in that way. But it's very hard to celebrate those wins.

Beth: Right. Okay. No. I think that's important because I think one of the things that we do within organizations is we spend an awful lot of time dehumanizing the people that are like the sysadmins that go through. And they just make sure that everything works.

And I think security gets that same — they are perceived in a very similar way except in a way maybe almost worse because they're the ones who are going to tell you, "No, you can't do that," as opposed to a sys admin that you go to.

And it's like, "Hey, everything broke. And everything is on fire. Can you fix it?" And they're like, "Okay. Yeah. It's 4:00 in the morning. Sure."

Ben Hughes: Or they like build things and like —

Beth: Toss it over.

Ben Hughes: — provide you with — well, like systems people provide you a platform to run your thing on. Or like they've enabled this growth. Like security hasn't stopped you doing your work in an annoying way.

And you're like, cool. Thanks for not making my life as annoying as you could have. But you've made it this annoying, which I see as more annoying. But the delta of annoying is not as much as it could be. And like no one celebrates that.

Beth: Right.

Ben Hughes: No one is like, "You brought in awesome things like Duo for 2FA rather than SecurID." And you're like, "Now, I have to find my phone," rather than, "Now, I have to find a small token that may run out of batteries in two years and may or may not have been lost somewhere in my house." So that's like actual real-world challenges, the integration of these things.

Beth: Excellent. So have you found any tools that really help make that easier? Or are you just going to drink a lot of gin?

Ben Hughes: Yeah. Just gin is the main tool.

Beth: Okay. Yeah. It's a good tool.

Ben Hughes: Yeah.

Beth: I find that, in product, it works pretty well. Okay. Fair enough. So I only have one more question that I think we didn't really cover of my highly prepared, talking to my cat of all the questions, which is, is there anything that I missed that you'd like to add?

Ben Hughes: Sure.

Beth: That's my favorite question.

Ben Hughes: Yeah. I just got back from sunny Las Vegas for DEF CON and Black Hat, the two biggest security conferences of the year. Well, one of them is a security conference. The other one is DEF CON. Throw some shade already. [laughter] See who I haven't upset in the industry yet. And then —

Beth: There's no fedoras in the room.

Ben Hughes: Okay. That's good. From that, there's certainly a lot of interest and hype in the newest bug or the coolest exploit. And there's not been as much focus on the, "Well, this is the coolest mitigation against this exploit. Or this is how you defend against these things," because part of how humans appear to work in that we quite like competition and pushing each other down and part of how marketing — bless them — works is you have to have the excitement and the hype.

But that means you don't focus on the actual useful things like defending stuff. You focus on the cool media thing like, in the exciting world of soccer — which we have in the rest of the world — no one can name five goalkeepers. But they can probably name five strikers. And you're like, yeah. That seems a fair balance.

So it's that kind of mentality. But I think slowly the industry is realizing that, if we actually want to make things better, that we can't just keep breaking stuff and going, "Found a hole in your stuff. You're an idiot."

You're like, "Cool. But you couldn't make this in the first place. So cool. But thanks for telling me about the vulnerability," and actually work out how to actually make things more secure rather than just — what's a polite way of saying ego contest? Ego contest.

And there are some cool bugs. I don't want to take anything away from security researchers and all those people who do things I could never even begin to do. But not all the time are they actually pushing the agenda of making things more secure. They're just finding cool bugs for fun, which is totally a cool thing to do. But — yeah. It may not be making the world a happier, better place.

Beth: Fair enough. All right. That's a wrap for our very first security podcast. Thank you very much, Ben Hughes. As I mentioned at the beginning, Ben — you can find on YouTube a number of his security videos and look for other podcasts that he's done.

Also, if you find him on the Twittersphere, you can find a link to his imposter syndrome and security document, which is very good. So thanks very much everybody for listening.

And thanks, Ben, for helping to talk about the more human side of the security world. It was much appreciated. And I hope that was useful for all of those listening.