To Be Continuous: Continuous Integration at Microsoft

In this episode of To Be Continuous, Edith and Paul meet up with Microsoft program managers Simina Pasat and Joshua Weber to discuss how continuous integration plays a role behind the scenes at Microsoft. Hear how they develop across platforms, use feature flags, approach user feedback, and more.

This episode of To Be Continuous, brought to you by Heavybit. To learn more about Heavybit, visit heavybit.com. While you’re there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Joshua Weber is a Microsoft program manager on the Mobile Center team where he leads development on the HockeyApp platform.

TRANSCRIPT

Paul Biggar: Okay, so, what is the thing that you like leastabout continuous delivery?

Joshua Weber: I could start off. Because it’s actually,I feel passionate about what I like leastabout CI, which is, I hate setup.

Every single time I’ve ever done a CI system,it takes me a day.It’s always a day.It always feel like it should take an hour or half an hour, because it’s always like, “Oh yeah, I know what I’m doing.” I wanted to just build and do the stuff,but it’s just it always takes way longer than I want.

Paul: This is literally why I started CircleCI.

Joshua: I would say that it’s by far, every time, my least favorite thing.It’s always getting in the way of coding.It’s always starting a projectand it’s always, I don’t get to start because, instead, I am installing agents,or trying to figure out a connectionor an authentication problem, or something like that.It’s very passionate for me,that it’s the setup costs.

Paul: The first time that I started marketing CircleCI,

the email that I sent out to our initial listwas, “Set up your CI in 20 seconds,”and that was massively effective.

Edith Harbaugh: Was it true, though?

Paul: In about 40% of cases.

Simina Pasat: How about the rest, of 60?

Paul: It took less than an hour.It was straightforward to get it set up.There’s no agents to install.We have the infrastructure as well.It was nice and straightforward.

Edith: How about you, what’s the thing you like least?

Simina: The least I like when there is a new version of Xcodeor a new SDK, and then it starts.You have to update the Mac hardware,you have to update the agent,then things start failing.Then if you have one app on one SDK version,and another app on another version,things might start failing even worse.So it’s becoming a huge mess.

Paul: I feel a lot of this pain.I feel like, as fellow CI providers,this is one of the nightmares.Your build machine or your build image has to update because the technology updates all the time.But then you fuck over the existing customerswho were perfectly happy with how things were working.

Simina: Yeah, and people expect in the next 24 hours, it’s there.

Paul: As soon as you make it.We had a process where building our Mac image in took three days and there was no waywe were going to get anything done in the next 24 hours.

Edith: Paul, now’s a great time for the gueststo introduce themselves.

Paul: As people who are not traditionallyin the Microsoft world, it’s funny to seehow different everything is in Microsoft land.But also sounds like quite a lot of it is the same.But from a mobile perspective,what is different in Microsoft world,than for us Mac lovers?

Joshua: I think the interesting take from me is, and I’ve been around in Microsoft for five years so I guess I can say that I’ve been on the inside,seen the transition, and I would say we really do feel that any developer,wherever you are, it is really like the motto,

When it comes to the mobile world,we evaluate the mobile world,we see where the majority of the developers areand that’s where we leave off.We always leave with the largest groups of developers.We’re always trying to find the largest value propositionthat we can provide to those developers.

I would say that hasn’t always been quite the waythat we’ve done decision-making,but it definitely has been for the last few yearshere at Microsoft.I would say that’s the part that I thinkis starting to come out at things like the Build conference,where it’s really starting to,you’re starting to see that in the actions that we take.

Paul: The CI/CD that you guys advocateand that your customers use,is that roughly the same as the rest of the ecosystem?

Simina: I think we took a really, very non-Microsoft-yapproach, when we started. Because we wantto meet the developers where they are today, and we said mobile developers are today on GitHuband are mostly developing iOS in Swift or Objective-C.

First iteration of the product, while you’re in preview,was connect with your GitHub repositories,build and distribute your iOS apps,your Objective-C, Swift apps and then we startedlooking at Android Java, at Xamarinand at other solutions as well.

Paul: Gotcha. And Xamarin is now part of that whole thingsince the acquisition?

Simina: Yes.

Joshua: I think we merged with the Xamarin team.It’s been really exciting to see, especially, I would say,how passionate the Xamarin community has beenas they’ve kind of folded into the Microsoft stuff. And the Xamarin solution’s fantastically incredible.

You can actually write native UIand then have it deployed. It actually bringsall this consistency across all your platforms,it’s really exciting to see the stuff.We’ve really had the excitement and maybe some new energy.

Xamarin was an external company and so I think they brought ina very different perspective to how we’re doingthe development.

So it’s really fun to work with those guysand partner up with them.

Edith: Yeah, I remember last year I was at Buildand they announced that Xamarin was now availableto everybody, and people were literally cheering.There’s so much excitement aroundthat amongst the developer community.

Paul: Xamarin is a C# solution, right?

Joshua: That’s correct.

Paul: It’s interesting that you folks were doingnon-Microsoft-y technologies and it was the externalcompany that brought in the C#-ness to mobile.Then Hockey was an acquisition as well, right?

Joshua: Yeah, HockeyApp was an acquisition.

Simina: It was in December 2014, I think.

Joshua: I think maybe two and a half years ago.

Paul: Was that before your time or ahead of that?

Joshua: It was actually right at my time.When the HockeyApp team, I actually joined directly with the team, when it merged in.That was really exciting too,especially with the popularity and how commonHockeyApp was.

I would say, especially for that iOS distribution market,they really focused in at the starting days.They were very much an early innovator, and that’s base distribution.It was really interesting to see this long historyof development and this long engagement with customersthat they had built up over time.

Simina: And for HockeyApp, it’s so exciting.Because most of their customers are people that we at Microsoft, until now,we didn’t have contact with.

Paul: Because they’re so small?

Simina: Because they’re small, because they’re just nottraditional Microsoft developers, they don’t use C#, there’s many smaller companies. And it’s just so exciting to work with them.

Edith: What do you think is the difference betweenthe traditional Microsoft developerand a non-traditional Microsoft developer?

Simina: That’s a good question.

Joshua: I think it has a lot to do with just the stack.That you end up, I mean, everyone talks aboutthe platform stack that you pick.There’s very, definitely a Microsoft stackwith a lot of tooling and a lot of richness.But it’s not the only option out there.

There’s a lot of other stacks,especially with the change in mobile development, to these brand new platforms for Android and iOS,where they were brand new platforms,nobody really knew at the start what they were doing.

I talked to the founder of HockeyApp, Thomas,and things like that.He talks about being one of the first peopleto have the original iPhone, and actually, his co-founder, Andreas, was developing an iPhone appbefore they had even published the APIs.Being there right at the thing,nobody really understood the new stacks.And there’s these entire ecosystems of toolsand stuff that have built up into those communities.

For a while the communities were very separateand maybe independent, and I think what’s been happeningover time is that these communities have been merging backtogether.

It turns out that we all havethe same problems to solve andthat innovations and the one set of tooling in our areas,something like CI or CD or these DevOps movements,it’s starting to bring these communities together.Where it’s not so much, you make one platform choice and then that means that you have seven other tool choicesthat just become obvious because that’s the platform you are.And I think they’re starting to become flexibility now, where you can go out and pick the best tool for the joband the tools support multiple platforms.I think that’s really great.

I think it’s bringinga lot more diversity to these different platforms,where developers aren’t just locked inbecause they’ve made a platform choice at the start,to all the rest of the toolingand now they can experiment and see someof the different perspectives that other people bring.

Simina: In fact,

Microsoft developer or non-Microsoft,in the end they have the same goal.They just want to make some really cool apps that people love. They want to have good reviews.They want to make money.In the end, everybody wants the same thing.

Joshua: They have very much the same problems we all hate.

Simina: The app expression for everybody.

Paul: From the perspective of CD in the mobile app space,I feel that there’s a lot of weirdnessspecifically around the App Store.You mentioned reviews and the factthat there’s the app review process,how does that effect continuous deliveryfor people in the mobile space?

Simina: It affects it quite a lot. That’s why we think,because you have to wait forever to get your app reviewed,not forever, but quite long,it’s really valuable to have a lot of iterations. And we guide our product a lot aroundthe “build-measure-learn” methodology.

We really think that once people start building the app,they want to test it with early testers,with their internal teams, maybe with alpha testers.They want to get feedback, they want to measurehow people are using it.And then at some point start doing deployments to App Storeand then get reviews and so on.

Even once they deploy to the App Storeand the app is out there, it doesn’t meanthat their build-measure-learn stops, right?They still are doing alpha testing. All the games are always having some betas out there.We guide our product a lot around just makingthese continuous iteration cyclesreally, really easy for mobile developersand just making them immediately, not have to waita few hours or a few days, but just shipping the appsas soon as possible in the hands of their testers.

Edith: I was formally at TripIt,which was a number-one travel app.It was something we were very serious about is,that you’re always iterating.

You try to get stuff in the hands of users,whoever they are, as soon as possiblebecause user feedback is everything.

Simina: I think, in the world of web apps,it’s much easier to do itbecause you have so many tools. But for mobile,you actually have to take the appand put it on their device,which makes it a bit more challenging.

Joshua: I always found that was the biggest paradigm shift for me, was always when you move from web to mobile.In the web world you own everything:you own the server that the web server’s running on,you own the back-end database systems.Ultimately, you could just reach out and SSH into themand actually tech them.

When you go into the mobile world,there’s so much of that experience now that’s just outside of your control, right?

The actual device running it is a devicein somebody’s pocket.You don’t have any influence over that,any control.

You can’t push stuff thereor the App Stores, where your delivering mechanisms,it’s not your own server you’re deploying to,it’s some external system, that’s controlledby Apple or Microsoft or Google, and you losea lot of that ability to control and influence in it.

I think, for mobile, the transition shiftwhere you’re going to have to start workingwith these external systems instead of your own systems,where you have so much more influence,results in a lot more thinking aboutwhat the experience is, trying to define those interfaceswith those external systems to how to work best

Paul: I’m not really sure I see such a stark difference.The way that I see it is that both mobile and webhave essentially two phases, I don’t want to say two phasesbecause that’s another thing, but there are two stepsin getting code in front of the customers.

The first is getting the bundle that includes that codeinto customers hands, and in that caseit’s the App Store or it’s deployment to the server.But the second phase is enabling the code,feature flags, obviously, what Edith’s company does.

Edith: Thanks for wearing your shirt today.

Paul: No worries. For once, you are wearing a LaunchDarklyT-shirt. It’s a thing we do. Never mind.If people are taking that strategyof, “the code doesn’t change until you flip the flag,”then the fact that there’s an App Store reviewor just code review on your web process, on your web deployment process,doesn’t really look all that different.

Joshua: Interesting.

Edith: That’s what basicallywhat my company, LaunchDarkly, allows you to do, is separate out deployment from visibility.You can push a build to the apps, to the App Store, and then selectively turn on featuresor turn them off if they’re performing poorly.

Joshua: It’s an interesting idea.You’re saying that by integrating inthe feature-flag-type capability,that a little bit return that control back towhere you can change the deployment to a little bit on your terms,as opposed, just the actual deployment to the store.It returns that a little bit to,it turns the same, I pushed the package to the weband then enable it, and I push the package to deviceand then enable it.

Simina: I think for the React Native world,it’s even a bit more easy. Because you have technologieswhich really do updates in the app, without having to modify the binaryand your code push, or other similar technologies,you just go in the app and you get a new updateand you had no idea that you had any update.You just have a new package.

Paul: I remember this. There was this giant fiascoa couple of year back, you might remember aboutthe Amazon Kindle app, that deleted all of your books.Amazon launched it for iOS and then had to getspecial Apple permission to launch a new versionof the same app, the next day and get a fast track through.If they had had the new stuff behind a feature flag,it would’ve affected a handful of peopleand then they’d turned it offand it would never have become a big thing.

Simina: That’s an approach we take with the productthat we are working on all the time.

We have everything, literally everything, behind feature flags.

Edith: Oh, cool.

Paul: Do you have features around helping customersdo the same things?

Simina: No, right now we have feature flagsto help ourselves to help customers.

Edith: I’d love to hear more about that.Do you have everything behind a feature flag?And what’s Microsoft’s process on that?

Simina: I think Microsoft’s process is very differentacross different teams. In our team,the way we run the product isthat we have a support chat window in the product itself.So people ask us for features and things they want to try.

Many times we just take one of the asksthat appears every time and we implement it. Then we ask the users that have asked for it, “Hey, can you give it a try? Is it what you expect?”Then we have a few iterationson that feature flag capability. Then onceit meets the bar, we just release it to everybody.If things go wrong, we have a way to make surethat we don’t delete everybody’s apps or such things.

Paul: What is the process when you decidethat that’s not going to become a feature?

Simina: Many times we just shelve it for a while. We just don’t release it.

Paul: So you have 100s of half-implemented featuresbehind a feature flag that touches one customer?

Simina: We didn’t start the product so long ago.We didn’t get time to get to 100s,but it’s very good for iterationsand for just finding out if what we are working onfor a few weeks, it’s somethingthat we should invest more months into,or if we just should call it a nice experiment for nowand re-discuss it at a later date.

Paul: It sounds like you don’t go backand delete those features.

Joshua: We do at times.

We’ll do a stage rollout sometimeswhere we’ll push a feature just to a few customers,especially since we have that Intercom techniquewhere they can actually talk to us.

Joshua: That way we can collect up a small set of customersthat are actually interested in this feature,and then we can deploy it out to those folks early.It’s like a little bit of a limited releasethat gives us a little bit of a timeto evaluate the initial response from those customers.

At times they’re just happy, and we roll it out fuller.At times they maybe feel like it’s not the right solution, and so we’ll just continue to iterate, usually, to get it.I would say it’s been maybe rarer for uswhere we’ve decided to abandon a scenario.I think that we tend do be pretty tenacious.

Once we take on a scenario for a customeror feel that this is a real customer pain point,I think we tend to be pretty, “Let’s keep at it.We’ll keep iterating, we’ll keep redefining the product.” It feels hard for us to give up on a user scenarioand just be like, “No, that’s not somethingthat we’re going to solve.”

Paul: It sounds like, in that case,the user scenarios that you start to implementmust be pretty fully baked at that point.

Joshua: I would say so. I think we usually feellike we’re pretty confident by the time we take ona user scenario, because it is pretty costly.

Paul: You mean by the time you start to actually implement it?

Joshua: Yeah, when we start actually implementing itand the resources, I think, like to feel like we’re pretty confident on that.

Paul: Do you have some phases before that,where you’re trying to invalidate your hypothesisbefore it becomes a user story?

Joshua: I would say that’s always part of our process.I would say it’s always somethingthat we’re trying to get better at.Making project decisions is difficultand trying to evaluate where you spend your resourcesis always a hardship.

It’s always something we strive to improve,but we do try to go through a phase in the startwhere we understand the right features to go afterand the right investments to make.We do things like customer interviewsand even paper prototyping customer development.We really try to do a lot of that customer engagementwhere we just talk to customers,understand their pain points.

Paul: What you’re saying is that there’s many placesfor a feature to die before you’re actually implementing.

Joshua: Yes, for sure.

Simina: It dies so many times.

Joshua: For example, we just tried something new yesterdaywhich is that we have a little mini dev daywhere we brought in a lot of the Build developersdirectly into the Microsoft campus yesterdayand then we just asked them questionsabout upcoming features, upcoming pain pointsto try to identify.

That’s a great waywhere we can try to filter outsome of those scenarios or pain pointsthat maybe aren’t worth further investigating.Or try to get a little bit more confidencein our own decisions, that the onesthat we are going to invest inare the right solutions for us.

E– Tell me about it, you laughedwhen you said you’ve killed many features. Do you want to?

Simina: With Mobile Center we started less than one year ago.We have HockeyApp and Xamarin test cloud before,so Mobile Center is the new generation.Obviously we have a lot of things to catch up withand we have also a lot of customers to talk to.I think we had some good learnings around thingswe thought would be useful.

For instance, personally, we wanted to offersome experience around unit test and we thought,“We can be really MVP and see what people are saying.”And we found out that the MVP is not enough for people.

Then we decided, “Okay, right now we don’t have resourcesto invest heavily in everything, what we should do. So for now, let’s shelve it for a whileand when we can invest in it fully,we should go with a full-blown solution.” Because just an MVP is not enough or people.

Joshua: Too minimal.

Edith: That’s always hard.

Simina: It’s not viable enough.

Edith: That’s the hardest thing as product managers.I love MVP’s, but what are they just sometimes too Mand not V.

Simina: Exactly.

Paul: Did you leave the un-viable thing, the feature in,and some customers are still using it?

Simina: We left it for some customersthat really wanted it and they still provide feedback.They said, for them, “It’s fine for now.”But we made sure not to roll it out to further customers.You don’t want to set the expectationsand then to take it away.

Paul: One of the places we end up, when that happens,we have it rolled there to a bunch of customersand then more customers hear about it by word of mouth or people tweet about it. And then it’s like,we’re in this awkward situation of, “Should wethen roll it out to another customer?” Especially if there’s migration involvedto the newer version that we want to do later,as it ends up being really fucking awkward.

Simina: I think now we have the benefit that we are in preview.We have still a lot of space to experimentand people are really excited that we experiment a lot, and they understand.They’re always, “Give me access to this,I want to try it out. It’s fine if you take it away,it’s just, I want to see.”

Edith: I did a webinar with Ed Blankenship,who runs another group. And in it he said,“Hey, I actually have all these feature flagswith new features, people were so excited!They were like, ‘I want to turn all this stuff on.‘”

Paul: I really like enabling, allowing users to turn it on themselves.

Edith: That’s what Visual Studio has now.

Paul: Oh, gotcha, nice.

Joshua: We’ve done that same self-enablementprocess with HockeyApp.We started in HockeyApp, what we were callinga “preseason program,” because it’s HockeyApp,so everything is hockey.Preseason was the early-adopter program,and I found it was very successfulbecause it allowed the customers to want to bewith the latest, who maybe have a little bit moreof a tolerance for some of the uneaseor the experimental-type features.

It gave them a way that they could self opt into being on the latest, and then seeing all the stuffand I found that by allowing your usersto self select themselves in that community,I think it actually worked out really well,that they can try out these features in advance.

Edith: I think, what I’ve found is, that makes them feela lot more community.

Joshua: Yeah, for sure.

Paul: I’m curious, how big is the community? And when you have one of these features,how many people are we talking, who enable it?

Joshua: For the preseason for HockeyApp,I think it’s a couple of thousand.

Paul: How many total HockeyApp users?

Joshua: I don’t think we usually share the total numbers.We’d say it’s much smaller,it’s a minority of the population,it’s not a major segment of the population.

Edith: Joshua, I love puns, so what are some other code namesthat you can share from HockeyApp?

Joshua: I think we have our internal environments.We also pick up the same thing, so we called them “training,” you know, you’re practicing.I think we had the preseason program.

Simina: We had “the rink,” which is the external web app. It’s rink.

Joshua: I think the command line tool that we were using is “puck.”

Simina: We have the logo, which is three puckson top of each other.

Joshua: They definitely tried to keep with that HockeyApp theme.

Simina: Do you have a goon somewhere?

Joshua: You know, we don’t.

Simina: In the Mac app, there’s a Zamboni.The logo for the Mac is a Zamboni for the ice.

Edith: That’s pretty cute.

Joshua: I really liked your idea about bringingthe feature-flag concept into that CI/CD.I think, at least for myself, I usually think of CI/CD,I got caught up in the build infrastructureand the actual deployment of the package.

But in reality, we should take a step backand really look at the whole value chain.Which is, you do a code commit as a developerto have a new feature, and then it needs to get intoyour customer’s hands and you want thatto really feed back into your cycle.

Paul: The separation of CI and feature flagsis a little bit unnecessary in a certain extent,that your one is an attempt to getting the codeinto production and you’ve got this branchin your Git repo, which is essentially the feature flag.But it’s an incredibly coarse feature flag.

If you have people who, I’ve definitely heard of peopledoing this, who do feature flags by Git branches,and it ends up that that isn’t possible for themto have multiple axes in whichthe feature flags are happening.I think it’s interesting, that the rapid delivery process,I think it’s sort of based on or sort of emerged from this ideaof continuous integration.The old meaning, before CI meant build servers,when CI meant, “We start a new branchand then we merge it as soon as humanly possible.”We call that feature flags now, I guess.

Edith: I guess.

Paul: CI name means build servers.

Edith: Thanks for joining us today.Do you have any final thoughts you want to share?

Joshua: It’s been fantastic talking to you guys.

Simina: It was really fun.

Joshua: I really like these different opinionsfrom different communities bringing everything together.

Paul: That’s wonderful having you here.

Joshua: The collaboration’s always exciting.

Edith: I loved hearing about how Microsoft builds software,because

it’s always fun to talk to the peoplewho are building software for people who build software.

Simina: It’s also fun for us because every team does it differently.If I talk with friends from other teams,completely different approach.Best practices are always there,but not everybody does the same thing.It’s always nice to talk to,even with people inside Microsoft.

Edith: Yeah, I think there’s this perceptionthat there’s a “Microsoft way,” but what you’re sayingis there’s more different teamsthat build in different ways.

Simina: It’s a huge company, right?It’s very difficult to converge all the peopleto the exact same thing.

Andrea does marketing for developer-focused companies. Before LaunchDarkly she worked at Terracotta, acquired by Software AG, and built the demand generation programs at open source in-memory computing company Hazelcast. Andrea is happiest when she’s swimming, karaoking or learning something new.