One year at Mozilla: SSL/TLS, MIG, Risk Management, Winter of Security, ...

It's been a year (plus a couple weeks) since I joined the Operations Security team at Mozilla. By and large, it has been the most exciting year of my - allegedly short - career.

SSL & TLS

A few short weeks after I joined, Edward Snowden and The Guardian published the first NSA revelations. To say that it shook up the crypto world is an understatement. It pushed us within OpSec to take a closer look at SSL/TLS support across the board. Several weeks and dozens of conversations with many domain experts later, I published Server_Side_TLS and cipherscan. It made the front page of HN and a few other sites. I quickly learned that the level of scrutiny the "internet" applies to Mozilla's words is waaayyy different than when I used to publish stuff on my own. The level of contribution is just equally impressive: people from all over the globe would send their input on how to improve the guidelines. Awesome feeling really, but also a strong sense of responsibility.The next step is to figure out if we can safely disable 3DES and RC4 from our ciphersuites. Michal Purzynski's work on NSM & Bro(zilla) is in the process of providing that answer, by flagging clients with CLIENT HELLO devoid of AES ciphers.

MIG

SSL/TLS regularly takes up some of my time, but my biggest and main project is Mozilla InvestiGator (MIG). When I joined, Joe Stevensen and Guillaume Destuynder had outlined the idea of a distributed platform for the detection of indicators of compromise, Something similar to Google Rapid Response (GRR), is what they had in mind. Having spent the last 7 years studying distributed systems, that part was easy to model. But I wasn't as comfortable with the remote forensic aspect of the project. In may, june and july, I evaluated GRR, Volatility and OSSEC, read tons of papers and articles, and eventually came up with the basic design of MIG: a distributed agent/server model, with strong security primitives, trivial to deploy and very reliable.

Now all I needed to do, was write code.

Like any reasonable computer scientist starting a new project, I went with what I didn't know: Go. I wasn't just attracted by the shininess of a new language though. The idea behind MIG is that we could deploy the agent very easily, with a simple curl https://example.net/mig-agent && sudo ./mig-agent . The agent would be a statically linked binary with all the configuration already built in. We wanted to avoid the provisioning tool wars, and be able to run it on systems managed by puppet, or chef, or manually, or not managed at all. When it comes to ease of deployment, Go has the best toolkit by far.

So I went and learned Go. The architecture of MIG is fairly standard: RabbitMQ handles the messaging between the agents and the scheduler, mongodb PostgreSQL stores the data, and the rest is built in Go. Humans are called Investigator, and they need to GPG sign their actions for the agents to accept them. Some critical actions may requires signatures from multiple investigators. Interactions are done through a REST api. Agents communicate asynchronously with the scheduler, allowing them to disappear for a while and pick up where they left off. Etc, ...

Around the beginning of this year, I opened the source on github: Mozilla InvestiGator Github Repository. At the moment, we are working on deploying MIG everywhere, and using it for compliance checking. Which is the third area where I have spent a lot of time over the last 12 months.

Risk management

Mozilla has been using Risk Management techniques for years, mostly during security reviews. OpSec didn't have a proper framework to quantify risk, so Guillaume Destuynder spearheaded the task of building one. When a project starts, we perform a rapid risk assessment. We look at risks in terms of image, finance, legal and operations. We quantify these risks on a four levels scale: low, medium, high and maximal.

Then we define security requirements. A high risk project would run on systems that comply with the high assurance level of the system security policy. We are still defining these policies. Take 6 people with 6 different opinions, and strong security background, and make them agree on a framework. It's going to take a while. But the results will be awesome.Saying that something must be compliant with a set of requirements is only one part of the story. The second part, the hard one, is measuring that compliance level. This is where MIG, and a number of other tools, come into play.

If a system is marked as medium assurance level, we know that its SSH configuration must not permit root login. MIG has a filechecker module that can match regexes on files. We simply use that against the SSH configuration file to look for /(?i)^permitrootlogin no$/, and return a compliance result true or false. Dozens of compliance checks on thousands on systems gives a lot of data, which is where Jeff Bryner and Anthony Verez's work on MozDef comes into play. We store all of that data into MozDef and render it with Kibana. For pretty graph, Michael Henry has, by far, the fanciest dashboard to date :)

In comparison, my dashboard for System policy compliance is a lot less sexy, but it's getting there.

Winter of Security

Writing code is fun. But it's even more fun to do it in groups! Across the 4 or 5 security teams at Mozilla, a few of us chat about random security automation stuff every tuesday morning. It's the Security Automation group. A couple months ago, we thought that it would be neat to invite students to work on our projects, so we created the Mozilla Winter of Security. The idea is to give students a cool project to work on, professors an opportunity to connect with Mozilla, and us, well, awesome security tools!We launched MWOS just two weeks ago, and so far the response and interest has been very positive. The deadline for applications is set to July 15th, so we will know then if MWoS has picked the interest of students. Until then, there isn't much to say, other than the fact that I'm checking the submissions spreadsheet hourly daily.

That is one year worth of work in OpSec! I did not mention the dozens of supporting tasks that we do for infrastructure and engineering teams across Mozilla. It's a bit like doing security consulting, but internally. I've had the chance to work with the teams who run Mozilla.org, AMO, Marketplace, Firefox Account, Sync, and more... If the number of projects that Mozillians work on at any given time makes your head spin, imagine what keeping it secure does to us :)

Next year will most likely be centered around the same projects. MIG will (hopefully) consume a lot of my time, and I'm looking forward to doing more work on TLS. Especially with TLS1.3 and HTTP2 being just around the corner! Stay tuned...