PagerDuty’s 3-Year Journey to DevOps Culture

You can’t drive forward if you’re looking only into the rear-view mirror, said Tim Armandpour, senior vice president of product development at PagerDuty.His talk at the recent PagerDuty Summit in San Francisco shared PagerDuty’s journey to DevOps, what lessons they learned along the way, and what you can learn from their experience.

Tim Armandpour, SVP Product Development at PagerDuty

In today’s business market of increasing velocity, companies need to simultaneously decrease risk.That requires a shift to continuous delivery (CD), and that’s where DevOps comes in.

PagerDuty started shifting to the DevOps culture it enjoys today in 2015. And the company has lessons to share with others. Be aware, he said, that it’s a long process and you have to be all in.It’s not something you do by halves. “If you don’t have 150 percent conviction, don’t bother.”

What works at one place may not work somewhere else.It’s not one size fits all, but there are lessons everyone can take from the PagerDuty experience.

“Stasis and stagnation are rarely successful,” he said. In order to get the full value of DevOps, organizational changes are needed.Talent is finite; if you’re not leveraging the talent you have, you’ll lose.

“Over the years we’ve had to make significant investments and changes in how we develop and operate products,” he said.And he’s not just talking about technology.

Of course, DevOps tools are needed, he said, but aren’t the most important thing.It’s about the people, and making teams work.DevOps is a marathon, said Armandpour.Treat it as a new mindset.Don’t get bogged down in titles or small things. And if you’re not moving towards DevOps, you’re standing still in today’s economy.

Culture Shift

For PagerDuty, there were two major conceptual changes that are core to the success of its DevOps culture. he said.These two shifts provided a platform for everything else the company did along the way.

First, PagerDuty shifted to seeing its internal infrastructure as a product offering.Around the same time, it moved to a customer-driven mindset. Everyone is providing a product. Whether the customer is external or internal, said Armandpour, they are all looking for value.

When managers started focusing on the new everything-as-a-product mindset, it changed their organizational decision-making and focus.“It was a game changer,” he said.

The second decision was to change up expectations for employees.In breaking down silos and moving to a Team First mentality, they decided that everyone, up to and including the CEO would be in the on-call rotation. This meant creating new definitions around what employees are expected to know.

A lot of cross-training was necessary, he said, but at the end of the day, that brings humanity and empathy to the entire workforce.And makes everyone fully aware of their product, how it works, and what to do when it breaks.Now, doing one’s first on-call rotation is celebrated as a rite of passage.

One of the biggest results of this change is that decision-making has been distributed.

“Those closest to the action are the best ones to make the decisions,” Armandpour said.In the world of continuous delivery, things are moving so fast and downtime can mean hundreds of thousands of dollars lost for every minute down, so decisions for fixes need to be made at the lowest level possible.

“Trust is fundamental,” he said.This also means that management has to make sure thateveryone is given the right information so they can make the right decision at the right time.

Creating a Team First Mindset

Learning is hard. The culture shift needs to be supported all the way through the organization. Communication, trust, empathy, andagile all helped make the change to DevOps easier, he said, because now it’s all about the team.

Changing Expectations

For management, it’s focusing on HOW. How do we get more leverage out of our people?Knowing everyone wants talented people, how do we provide a work environment that makes our talented people want to stay?

Leaders need to focus on the human side of breaking down silos, including explaining what the individuals are gaining.“We told them, it’s not about you doing less in your job,” he said, “but a lot about helping our organization elevate to another place and realize our full potential.”

“DevOps means giving a shit about about your job enough to not pass the buck.DevOps means giving a shit about your job enough to want to learn all the arts and not just your little world.” — John Vincent, DevOps Engineer, ReminderMedia

It’s not possible to over-communicate, he noted.“Developers do well the more they understand the concept, and ‘why’ is it important.”

Developers now had to understand infrastructure and the full stack and what’s under the hood.Just being a driver didn’t cut it anymore.

They had to embrace operating their own code.

At the same time, Operations had to make things automated, easy, safe for all.This meant shifting from the mindset of processing tickets to one of empowering others to do their job.This change was tough, he said, but critical, so the operations teams didn’t fall into job preservation mode and stick to old patterns.

Your expectations have to change, both for managers and individuals, said A. “Your job is no longer to process 10 tickets a day, empower 100 engineers to do their job better.”

Knowing we’re all in it together has been really helpful.

What to Plan for

The three main areas needing budget attention for the transition to DevOps for PagerDuty were attrition, training, and decreased reliability during the initial shift.Armandpour budgeted for each accordingly.

The management team knew that the transition would necessitate retooling employee mindsets across the company.The biggest stumbling block for a lot of the people who eventually self-selected out, he said, was the required on-call rotation.Not everyone can roll with change.Expect some people to self-select out and budget for attrition.

For companies going all in, he suggests planning on losing 10-15 percent of your workforce. PagerDuty had 10 percent of its workforce decided to leave instead of making the cultural change. It was best, he said, to wish them well and send them on their way.

The second major component was budgeting for a dip in reliability.Expect that in the beginning a lot of stuff is going to break, he said.Prepare for that. Armandpour built that learning curve into their expectations and their budget.

The last major change was a dramatic increase in their training budget.Since the company broke down the silos and required everyone to be on call, all team members needed to be cross-trained.That took a while.They added management and leadership training, so their supervisors and managers would have the skills to guide their people through the shift.

Ongoing, agile training is held quarterly. PagerDuty has now customized their own curriculum and budget of about $40,000 annually for agile training.Another training investment has been in behavioral interviewing coaching since the DevOps culture is so much more than technical knowledge.

Ongoing, every engineer gets $3000/yr to apply to whatever training they want.They do have to justify how their training is going to help the company, but are otherwise free to learn what they wish.

These investments ensure that their employees are cross-trained and keep up with the latest changes, which increases employee satisfaction and contribute to PagerDuty being constantly rated as one of the best places to work.