Today I finished up work I’ve been preparing most of this week to update our linux build machine software.

A few new software installs, as well as a few software upgrades.

Most notably is an upgrade to our buildbot code in production here (to match what is currently in use at Mozilla for Firefox). [special thanks to dustin for his help getting me rpms and files as mozilla prepared them for their own use]

There should be no issues, since most of the changes were in the way our automation is setup not in what actually produces/creates builds. and this work affects all branches. (SeaMonkey 2.4 final is already built, so did not affect that).

If you see anything that seems broken to you on linux, or any weird issues, please let me know by filing a bug (blocking Bug 687797), or finding me on IRC.

Due to the relatively large quantity of reports we are getting, even with my comments in newsgroups, I think I should post this wider.

SeaMonkey 2.3.2’s download internal files, internal version, and all else of 2.3.2 except our website and files at our download directory location are indicating that it is SeaMonkey 2.3.1. (details in Bug 683473)

There is nothing to worry about there, we did properly include the DigiNotar cert block, as did Firefox 6.0.1. We are however also getting ready to release a SeaMonkey 2.3.3 release for a related Gecko issue, so we will correct our version number with that release, and not bother with any more updates with SeaMonkey 2.3.2 (See the bug driving this respin of Firefox at our bugzilla instance)

We are sorry about all confusion, and any issues this may have caused.

From my project leader point of view, as needed I deployed myself as an extra developer to speed things along. In the debrief, Luke called this the “glue person” role. He said having a glue person allowed everyone else to get into the zone and do lots of design work and big blocks of coding–everyone else on the project doesn’t have to worry about random bugs and and other small technical issues because the glue person picks them all up.

That is a quite astute description of WHY this type of contributor is such a blessing to any large project, especially one like our JavaScript engine.

The SeaMonkey Project also has someone filling this role, quite helpfully. Surprisingly role is NOT being filled in our case by our Former Project manager (Robert Kaiser) nor myself (my duties primarily reside in Release Engineering) and also not within our current SeaMonkey Council (The primary “Project Management” group) though many of the above do varying tasks, of course.

For us the person the “Glue Person Role” resides with, came to us a bit over a year ago, looking to help. He then met us at our Face to Face meeting in October, in Vienna Austria (invited due to the value we all perceived in his contributions, and the future value in having him come)

In our Bi-Weekly IRC meetings his Bug Fixed list is frequently very very long, and he actively seeks out bugs to fix, mostly choosing from our “Good First Bug” list. I’m coming to call him a jack-of-all-trades myself, even though we all still help with hand-holding every now and again.

I’m sure being a “Glue Person” was not his intention, but his efforts here are greatly beneficial to us in keeping SeaMonkey progressing at the pace we need it to, and ensuring that the minor annoyances don’t file up too high for most of our users.

So after all that,

THANK YOU EDMUND WONG![ewong on irc]

Your work is invaluable and I hope you keep it up. And for those of you reading this, if you always wanted to help, feel free to ask around a project you would like to help with. Being a glue person may not feel glamorous, but it is MUCH more helpful to the rest of us developers than you would believe.

To address leading developers on any project around here, realize the “Glue Person” is also a great reason to spend a bit of extra time helping new people to learn the ropes, or a bit of vocal “well you should peek here” to fix any relatively simple bugs someone wants to help with. We need more people like ewong across all our projects!

What about versions?

we created a comm-2.0 buildbot tree, which uploads builds based off of mozilla-2.0 to the latest-comm-2.0 directory on ftp.

Made builds which are based off of mozilla-central now listed as version 2.2a1pre (Our next version will be based off of Gecko 2.0)

The builds in comm-2.0 will continue to be SeaMonkey 2.1b3pre (and 2.1 final upcoming)

FAQ

I am a nightly tester, what version should I expect?

If you were testing a 2.1 build, you will automatically be upgraded to a newer 2.1 build; this does mean that from the point we flipped the trunk switch for version numbers, you may have a failed partial update; Do not worry if so though, as it will fallover to a complete instead.

But, I want to test nightlies on 2.2a1pre (Gecko 2.0.next)

If you want to follow the extremely undertested mozilla-central versions you can download a 2.2a1pre build from the latest-comm-central-trunk dir on ftp, and your updates will continue along that path. Do note, that we know of a few existing bugs on this configuration already, however both ourselves and the Thunderbird team are concentrating our efforts into the Gecko 2.0 based builds.

I am a local developer and want to build/checkout for SeaMonkey 2.1

You will have to pass to client.py the –mozilla-repo option for the stable mozilla-2.0 builds, we left client.py checking out mozilla-central. The recommended way is to rename the existing mozilla directory to something else (.mozilla-trunk is my recommendation) and then run client.py co –mozilla-repo=”http://hg.mozilla.org/releases/mozilla-2.0″

I am a developer, wanting to checkin what do I care about?

For all intents and purposes you can checkin to comm-central based on the status of the SeaMonkey2.1 or Miramar (Thunderbird 3.3) tree’s the SeaMonkey and ThunderbirdTrunk tinderbox trees are nolonger blocking landings. Also watch SeaMonkey2.1 and Miramar trees for tree-rules/status primarily.

I am a localizer, what repository are you using for SeaMonkey2.1 builds?

We are currently building l10n repacks with the releases/l10n-mozilla-2.0/{locale} set.

It has not yet been explored if we can/will do a l10n-central<->releases/l10n-mozilla-2.0/{locale} merge for seamonkey/thunderbird specific locale files yet, we will keep you informed.

I have another question, or am confused about something.

We’d love to hear from you if this is the case, please see the newsgroup (or mailing list) mozilla.dev.apps.seamonkey. Or just comment here and I’ll respond.

Ok, this week has been very busy for me RelEng wise, with some good, bad and ugly stuff.

The Good

I upgraded all our Windows slaves for the DirectX SDK, enabling us to ship the redistributable parts of that with our binaries, allowing most of our users to have GPU acceleration just liek in Firefox, where they normally would not have; also enables us to build ANGLE a critical part in that acceleration. — Mostly benefits our trunk/SeaMonkey 2.1 builds.

Upgraded our [windows] slaves to HG 1.7.5, enabling me to work on Bug 643324 which enables HG_SHARED support on our Windows Machines. My work there only enabled the use of share for the larger repo’s we have; namely comm-* and mozilla-* and left LDAP/etc. pulled as normal. (mac and linux coming soon). The idea and the basis for this was due to the work Chris Atlee performed on try recently, read his blog post for more information and why this is so good!

Spun up the oilspill builds for 2.0.13 [ftp] in line with the upcoming chemspill releases for Firefox 3.5.18. (I have not yet announced these builds yet, but I should have a call for testing up within 24 hours, feel free to jump into it with this blog post though)

The Bad

While doing the hg upgrade on our slaves, I created the hgrc’s I needed using the slave-side windows notepad; Forgetting of course, that notepad saves in Unicode by default, including a BOM; which our Hg was unhappy with, breaking some builds. At the same time I did not save some of the |.hgrc| files correctly, and they got windows-translated to |.hgrc.txt|, cleaned that up swiftly.

While preparing for the 2.0.13 release, I updated the configuration file locally, but forgot to specify the ssh://* repo url when I went to push, so my local push failed. I did not notice and reconfigured our build master and triggered the buidls anyway. This caused the already-on-ftp 2.0.12 build1 builds to get overwritten along with the support files. And delayed me having 2.0.13 ready sooner.

The Ugly

Because of the bad mistake with 2.0.12, I made matters worse. When I went to start to sync in from the pushed-to-release files, I clobbered in the wrong directory on stage (ftp), which meant I cleared the WHOLE build1 dir of SeaMonkey 2.0.12, causing me to miss some intermediate files that are needed by automation (I did lose the crashreporter symbols too, but those are unneeded by our automation at present).

Due to clobbering those files, I tried recreating them by hand, after a few hours of re-syncing in the necessary files as they should appear in that folder. I created the BuildID, but sadly, I used the abbreviated buildID (YYYYMMDD) rather than the full one (which varies by OS/Build here). The reason that matters is that the 2.0.13 automation, when generating/verifying the updates writes that build-id as specified on ftp to some files needed for sanity-check automation only and then uses them to download and test updates. So I then had to go back in and update those values correctly, update the config files for that automation step, and re-run.

I have not yet managed to have a trouble-free release since I took over, from “GO”->”Release”, here is hoping for next time.

Chris AtLee has been doing A LOT of good lately, and I just have to call him out on it, with lots of praise!

For everyone: faster try builds, nothing like speeding up the already lengthy try results by 30 minutes or more!

The other thing I REALLY want to praise him on, which many people do not realize, for the UpComing SeaMonkey 2.1 beta 2 build I had been working on, he helped me troubleshoot a buildbot issue, very hands on (step by step debugging) which lasted over 3 total hours. This was of course outside his duties as a Firefox Release Engineer, and in his spare time.

His help was invaluable in getting our release (based off of Firefox 4.0 beta 11) complete in a timely manner. And the help we routinely receive from everyone on the Mozilla Teams never ceases to amaze and astonish me!

So a heartfelt and big THANK YOU is in order for Chris and the entire Mozilla Staff for this one!

SEAMONKEY: Continuing to do the impossible.

SeaMonkey has not “fake” any other UA for browsers unless the user installed an extension or changed a pref to make that happen. The reasons for this were vast, mostly in the eye of not doing users and websites a disservice by lying.

That all has just changed with tonights nightly. [Bug 591327] The truth is that websites have increasingly been found to be broken with JUST SeaMonkey. Everything from some sites just downright breaking, (with XML Parse Errors), to some simple CSS Changes, to other websites treating unknown UA’s as a “mobile” version of the website, etc.

These issues we have increasingly felt did more disservice to our users than what we had established in the past. Of further note, is that these websites, due to Firefox’s popularity seem to frequently NOT be broken when and if we Added “Firefox/” to our UA string. Camino as one example has long had “Firefox” added to their UA because they felt it helped their users best.

The Bottom Line, we have toggled the (new to Gecko 2.0) pref by default to enable “compat mode” in the UA (which appends “Firefox/” to the UA string we send to websites), we feel that most users will not want, nor have a need to change this pref. But we have provided Pref UI under “Advanced” for the idealists out there.

Given the fundamental-to-the-goals-of-the-project change, this was voted on and approved by a SeaMonkey-Council vote as well, incase anyone is wondering.

… And why the tree is CLOSED.

Late Friday the 4’th, of June; I was looking to checkin a series of patches that I had previously pushed to the try server and got it to pass all tests. This series was a large block of the checkin-needed work in Bugzilla.

Now being a person who hates (within reason) merge’s in the history of m-c I went ahead with a |hg pull --rebase| for my MQ Queue full of all those patches. This to move everything up to tip, help stave off any .rej files from failed (trivial) hunks, etc.

So What Went Wrong?

It turns out there is an existing bug in Mercurial, which has been fixed in Mercurial Version 1.3 and newer. Of course this particular bug was Footnoted on MDC as well.

This problem is compounded by the fact that MozillaBuildversion 1.4 (which is the latest as of this writing) ships with Mercurial 1.2, a version which still contains this bug!

I advise anyone who has push access, or even better who USES hg rebase to upgrade their Mercurial Version TODAY. [Note, MozillaBuild will not pickup the install directory by default, you’ll need to change the PATH yourself, I did it for myself in my .bash_login]

For MozillaBuild, my hope is that Bug 557210 can be fixed sooner than later, and that we can release a newer MozillaBuild package to fix this. Within the next 24 hours, I will look into getting that bug fixed, as repentance to making this error myself.

The End?

Boy do I hope so! I hope this is the END of any all errors or mistakes relating to this type of history snafu. Not only does this hurt Hg Annotate (significant for this directory), but it would hurt the electrolysis landing as well (as there are non-trivial changes to netwerk/ that if this was fixed right would make it relatively trivial to merge in). so long story short I AM SORRY TO EVERYONE.