In picking buildbot as our tool we were improving vastly on the decade old technology we had at the time (tinderbox) which was also written in oft-confusing and not-as-shiny perl (we love to hate it now, but it was a good language) [see relevant: image of then-new cutting edge technology but strung together in clunky ways]

As such, we at Mozilla Release Engineering, while just starting to realize the benefits of CI for tests in our main products (like Firefox), were not accustomed to it.

We were writing our buildbot-related code in 3 main repositories at the time (buildbot-configs, buildbotcustom, and tools) all of which we still use today.

Fast forward 5 years and you would have seen a some common antipatterns in large codebases… (over 203k lines of code! ) It was hard to even read most code, let alone hack on it. Each patch was requiring lots of headspace. And we would consistently break things with patches that were not well tested. (even when we tried)

It was at a workweek here in 2013 that catlee got our group agreement on trying to improve that situation by continually running autopep8 over the codebase until there was no (or few) changes with each pass.

Thus began our first, attempt, at bringing our processes to what we call our modern practices.

This reduced, in buildbotcustom and tools alone our pep8 error rate from ~7,139 to ~1,999. (In contrast our current rate for those two repos is ~1485).

(NOTE: This is a good contributor piece, to drive pep8 errors/warnings down to 0 for any of our repos, such as these. We can then make our current tests fail if pep8 fails. Though newer repos started with pep8 compliance, older ones did not. See List of Repositories to pick some if you want to try. — Its not glorious work, but makes everyone more productive once its done.)

The one agreement we decided where pep8 wasn’t for us was line length, we have had many cases where a single line (or even url) barely fits in 80 characters for legit reasons, and felt that arbitrarily limiting variable names or depth just to satisfy that restriction was going to reduce readability. Therefore we generally use –max-line-length of ~159 when validating against pep8. (The above numbers do not account for –max-line-length)

Around this time we had also setup an internal only jenkins instance as a test for validating at least pep8 and its trends, we have since found jenkins to not be suitable for what we wanted.

Stay tuned to this blog for more history and how we arrived at some best practices that most don’t take for granted these days.

I spent a few minutes a week over the last month or two working on compiling a list of Release Engineering work areas. Included in that list is identifying which repositories we “own” and work in, as well as where these repositories are mirrored. (We have copies in hg.m.o git.m.o and github, some exclusively in their home).

While we transition to a more uniform and modern design style and philosphy.

My major takeaway here is we have A LOT of things that we do. (this list is explicitly excluding repositories that are obsolete and unused)

You’ll notice a few things about this, we have a column for Mirrors, and RoR (Repository of Record), “Committable Location” was requested by Hal and is explicitly for cases where “Where we consider our important location the RoR, it may not necessarily be where we allow commits to”

The other interesting thing is we have automatic population of travis and coveralls urls/status icons. This is for free using some magic wiki templates I did.

The other piece of note here, is the table is generated by a list of pages, using “SemanticMediaWiki” so the links to the repositories can be populated with things like “where are the docs” “what applications use this repo”, “who are suitable reviewers” etc. (all those are TODO on the releng side so far).

I’m hoping to be putting together a blog post at some point about how I chose to do much of this with mediawiki, however in the meantime should any team at Mozilla find this enticing and wish to have one for themselves, much of the work I did here can be easily replicated for your team, even if you don’t need/like the multiple repo location magic of our table. I can help get you setup to add your own repos to the mix.

Remember the only fields that are necessary is a repo name, the repo location, and owner(s). The last field can even be automatically filled in by a form on your page (see the end of Release Engineerings page for an example of that form)

Reach out to me on IRC or E-mail (information is on my mozillians profile) if you desire this for your team and we can talk. If you don’t have a need for your team, you can stare at all the stuff Releng is doing and remember to thank one of us next time you see us. (or inquire about what we do, point contributors our way, we’re a friendly group, I promise.)

This post does not attempt to elaborate on that specifically too much, but it’s more to identify some issues I hit in early testing and the solutions to them.

Theme

While I do admire the changes of the Developer Edition Theme, I’m a guy who likes to stick with “what I know” more than a drastic change like that. What I didn’t realize was that this is possible out of the box in developer edition.

After the Tour you get, you’ll want to open the Customize panel and then deselect “Use Firefox Developer Edition Theme” (see the following image — arrow added) and that will get you back to what you know.

Sync

As a longtime user, I had “Old Firefox Sync” enabled; this was the one that very few users enabled and even fewer used it across devices.

Firefox Developer Edition, however, creates a new profile (so you can use it alongside whatever Firefox version you want) and supports setting up only the “New” sync features. Due to creating a new profile, it also leaves you without history or saved passwords.

To sync my old profile with developer edition, I had to:

Unlink my Desktop Firefox from old sync

Unlink my Android Firefox from old sync

Create a new sync account

Link my old Firefox profile with new sync

Link my Android with new sync

Link Dev Edition with new sync

Profit

Now other than steps 6 and 7 (yea, how DO I profit?) this is all covered quite well in a SuMo article on the subject. I will happily help guide people through this process, especially in the near future, as I’ve just gone through it!

First some brief Background, Mozilla Releng has our code in a *lot* of repos, most being in Mercurial (a few other needs are in git or svn, but those are very rare relatively). I also do work for SeaMonkey which has needs with m-c, m-i, m-*, c-c, c-* etc. And needs with l10n…

I personally manage all my patches with MQ. Which presents a problem for me, “keeping track of it all”. I used to try keeping open bugs, but thats hard with releng because while a bug may be open, we tend to have a good handful of patches attached to it, for various repos, and they need to land in certain orders sometimes.

Other ways I’ve tried to cope have been with landing as soon as the review comes in and avoiding writing patches for parts that need to land later until the first parts are landed/deployed. I found that method encompasses unneeded end-to-end times on bugs, and unnecessary context-switching.

To curb that I wrote a mozilla-build (bash) script [in ~/.bash_profile ] that sets an alias `patchset` that I run, and it works!

It especially works because I keep my code in /c/Sources/hg/* some repos are multi-levels deep, so this code could/should be improved or at least edited for your uses, but without further ado, this is how I manage my patchset (again note, all my work is in Mercurial, I do convert my stuff over to git/etc as needed though):

Stats from Google Play for “Firefox for Android”

Before we begin with the data, I need to clarify something readers may not be aware of at first glance, “What is an Install”

Install in this context is any currently active device which has Firefox installed on it. It does not actually indicate frequent use.

Installs by OS – Release (org.mozilla.firefox)

Yes, you see that right, this is 81.4% of our GA users on some version of Android 4.0+.

Installs by OS – Beta (org.mozilla.firefox_beta)

Our beta audience is pretty similar with 83.59% on an Android 4.x or higher.

Installs by Chipset

Selecting by Chipset is a bit harder, since to do so we have to take a factor of how Releng does our Play Store releases (different buildID’s to factor to different chipsets). I am doing this by a feature of the play store, namely “Export as CSV” which gives buildID info

So with that in mind, here is the data:

Arm V7

Arm V6

x86

Firefox Beta

96.19%

0.90%

2.91%

Firefox GA

98.61%

1.39%

N/A

The caveats to note is that I only gathered data from today’s Google Play installs, and I aggregated all installs over all versions, even ones that are multiple years old. We also do not have x86 released officially yet, so we only have beta users using that version.

Coming in Part 2:

Where does Mozilla Invest build and test resources for Android?

How does this compare to Mozilla’s Testing Infrastructure?

¹ – This post as-is is indeed intended to be data without analysis/commentary. I don’t feel I’m greatly suited for the latter compared to other people possibly reading. In Part 2 I intend to show some correlaries-as-data to what we are doing inside Release Engineering as it compares to our users at large. I’m currently hoping to also write it devoid of any assertions/commentary.– The underlying reasoning here is to help spur thoughts and commentary in others in order to further our mission using data, while at the same time without inserting my own opinions or biases into what I am presenting with this 2-part series. I do not yet know if I will do a commentary piece referencing these posts or not, and I may do so, I just don’t yet plan to.

I am looking for links to all photos taken at summit locations, selfishly looking for ones of me, but while I’m doing this I had heard from IRC that there was a suggestion of “one big album” of pictures on a by-location basis.

So I’m offering to do a few things:

Gather said list of pictures, be it albums or individual pics

Inform anyone I personally recognize from each picture that they are in them

We have had a hardware error in the systems that allow us to reliably generate the release — Without these systems anything we create will be of unknown quality/stability.

In order to meet our own quality and stability requirements we are NOT releasing SeaMonkey 2.18.

While there is a chance we could have these systems back up in time to do an intermediate release (say something corresponding to a possible Gecko 21.0.1) we can not promise nor plan for it at this time.

We are actively working on repairing the system and its data, once that is complete we will go forth with a new BETA based on the SeaMonkey 2.19 train, and we expect to release SeaMonkey 2.19 on time, on June 25’th.

Yesterday, we were given the confirmation that we are now able to turn those tests back on, and we have done so. You should now see Fedora32 tests on all your Inbound, Central, and Try pushes, as well as any twig/project-branches.

Since our last conversation a lot has changed with me (and Mozilla!) I was a contractor since March, and just got my official offer to come on as a “real” employee. I will be detailing my journey in the community to a full time employee through a series of upcoming blog posts. (now don’t get me wrong, I’m still working on SeaMonkey in my free time too)

To also mix it up, I intend to throw in many blog posts about the state of our Mobile Infrastructure/Tooling, and where we intend to go from here. However I need ideas on posts! So to get me started, I’d like to enlist you, planet readers, to send me questions/topics you would like to learn more about when it comes to Mobile Automation (in Release Engineering) that I could blog about for you. History Lessons, Future Direction, Current Setup, Whatever!

Send your Topics/Questions to me at callek+blog@mozilla.com I cannot promise I will address every question/topic requested, but I can promise to try.