Sustained Engineering: Patch builds

We’re delighted to announce a change to the way we will release builds. Previously all bug fixes have been rolled into publicly released updates of the editor, shipped and published by the R&D developer team. As the complexity of the product and the organization grows, it gets increasingly difficult to maintain all these in one place. This has meant that bug fixes have taken longer and longer to get into the hands of our customers. We don’t like that.

In January we merged the QA and Support teams into one organization with me (QA Director) heading them. The main purpose of this merge was to enable us to do sustained engineering on our growing number of versions, such that we can better handle the issues our customers face in current released versions. By “sustained engineering” we mean working hard to improve the reliability of a currently shipped version of Unity, and not making fixes in future versions which can take time to complete, pass through alpha and beta cycles, QA fully, and ship. Effectively it is an expansion of the responsibility of the support team to also be able to fix bugs, backport bugs and release them directly to customers as patches.

Since the support team is not of endless size and the risk of making code changes on a released version is high, we are going to be very careful about which bugs we will fix. Pieces of the puzzle include the severity of the issue for the customers, the amount of customers affected, how long it will take to fix and a slew of other factors. The bottom line is that we will not be able to, or even want to, fix everything everyone asks us, but we will be able to do more than we do today and do that more often. Customers affected by annoying bugs should see these problems resolved much more quickly.

To be able to handle this, we have recently hired a release manager, Jawa, to handle the release train for these patches. He will handle the communication internally between all the teams in Unity and also handle the release procedure for each of the patches and update the forums on releases.

When we say patches, it is actually a full release of the entire editor, with all runtimes. The complexity of Unity forces us to do that. However, if you are hit by the issues we fix, it will be available on our forums for you to download, just like any other version of Unity you use. The main difference is that we will distribute the patches through the forums and we will NOT enable the editor update check on them. A patch will have a version number ending in “..pX”, so for example, a patch for Unity 4.3.8f1 will be called 4.3.8p1. This patch will have a small number of bugs fixed. These bugs will be listed in the release notes for that version. Then a new patch will be generated, which will include these fixes, and newly fix bugs. This will be 4.3.8p2, and so on. Once we have a set of patches ready, we will roll them up to a new “..fX” (f for Final) release, eg. 4.3.9f1, and that will undergo full regression tests and be enabled through the editor update checks for everyone. Note that we expect only customers affected by reported fixed bugs will want to migrate to patched versions of Unity.

To really kickstart this journey, we have gathered support engineers, field engineers, QA, build engineers, infrastructure engineers and release managers in Brighton this week. A total of 28 people have joined in a week of learning the processes, doing the actual fixes and shipping them. I will have to caveat that the focus here is on learning a very hard process of getting the code done right, it is NOT about having large quantity of fixes, so the first few patches will be very limited.

@peter: we can’t go through the normal QA cycle as fast as we want to release patches, so we have decided to release more often to a smaller group and then roll them up to full releases. And then there is the matter of pushing the installer to the world. It has some very real cost.

“we expect only customers affected by reported fixed bugs will want to migrate to patched versions of Unity”

It’s encouraging that you’re going to start releasing patches, but why would anyone want to run an old buggy version where they’re going to come up against bugs that have been already patched? Just release new versions more often. Maybe have the odd LTS release for customers who need the “stability” of an old buggy version?

Here’s an issue I came across today, or more precisely, the response to someone with the same issue:

“Unfortunately, retrieving the collider bounds for 2D didn’t make it into the 4.3 release however it has already been added and will be available in a future release.” — Unity Developer, 6 months ago.

It’s very frustrating to have to find workarounds when the solution is lost in a QA hole. Please don’t be half-hearted about releasing patches. Why deliberately bury them away in forums? Just make more release, more often.

Richard Fine nailed it perfectly. These builds are only spot checked for the specific bugs, so no manual regression test are run on them. All the automation is obviously green on any build we ship, but that is not the same as a fully QA’ed build.

@wonderer: Since I’m QA director, I do have a passion for stability and Unity has a very large QA department because we take the seriously. This is a move to take the stability issues into hands that are the closest to our users, namely support.

This is awesome. I’m really glad you are doing something to address this. It sometimes takes far too long for critical bugs to get fixed because they have to go through such vigorous testing along with the rest of that point release.

If you want to be notified of new patch releases, you can use the “Thread Tools >> Subscribe to this Thread…” command, which can be found just above the first post on the page. This lets you tell the forum system to send you an email whenever a new post is made to the thread.

Note that you do have to specify that you want to be notified by email; the default setting is to notify you through only your forum profile / control panel.

@Gregory Pierce: I think the reasoning is: in the general case you should not upgrade the version of Unity you’re using to these patch releases unless you have a particular problem that the patch release fixes. Every little change to Unity has a risk that it might have broken something, so if you use a patch release when you don’t actually have any of the problems it fixes, you’re risking breaking your project for no gain – and why would you do that?

My guess is that these patch releases will have a much shorter QA cycle than ‘official’ releases, in order to get them out quickly. If you’re blocked by a bug that they’ve patched then you may want to take the risk and grab the patch, but if you’re not affected then you’re better off waiting until they do a bit more QA and roll it out as an official point-point release (4.3.8 etc).

It’s good that you are allocating more people to fix bugs.
I wonder if you will ever realize that users prefer a stable software to work with over AAA features that 95% of them are not going to use.
(Yes, 2D was important, but maybe it would’ve been released 2 years ago with proper prioritization. I wonder if it was ever added if you didn’t feel a threat from GameMaker in that field)
What @Scott Richmond mentioned shouldn’t happen with a respectable software company.

Glad to hear that you guys have done this, but I’m curious as to why you wouldn’t want to expose the patch builds via the editor similar to how one can get on an “unstable” or patch tree of Chrome or other technologies. While I visit the forums from time to time, I don’t do it often enough to rely on for looking up patch builds.

@peter: It might fit, but we enter the land of what is possible. PSM is handled by a small team on a super specialized platform, so our SE engineers can’t fix it ourselves. But the PSM team can inject fixes into patch releases, if they decide it is important enough.

I’m a bit confused. Does that mean you are going to patch the PSM build as one of the criteria you list is the severity of the bug and the number of customers it affects. Well severity is critical (messed up rendering) and customers affected er all using the PSM build!

This is a great step forward. Though I still continue to be shocked at just how bad the bug support from Unity is in general. Our team have been facing a critical OSX bug for 4 months now for our commercial game, and they barely even take the time to review let alone confirm the bugs’ existence.

Wow. You all should be ashamed of yourselves telling Unity their new bug-fix release strategy is a bad thing. Anyone ANYONE who knows anything about releasing software professionally knows that what is being described here is a HUGE step forward for reliability and maturity of the product. And FYI: It’s a “patch” because the bug is fixed by writing a “patch.” That’s a thing we software engineers know about. Perhaps you-all should learn something about distributed software development and release strategies before you decide you think Unity is inept? I say, bring on the bug-fix releases. Thank you Unity, for understanding how important these are to us. If I’d been able to get companies like Softimage and Autodesk to adopt that kind of bug-fixing strategy, it would have saved a ton of time and money over the years at the companies I worked for.

Sounds good :) as an Avid iOS, Android and PSM developer i’m really happy to hear this! As various programmes in the past *cough Xcode cough* have often delayed development by weeks >_< so yeh, ! nice work and good job, keep it up guys! Best Game Engine ever!

@TJ Unity is a big company made up of many teams. I read that reply as if he is talking about what his team can and can’t do and Graham Dunnet helped answer the questions on the broader scale of the company direction.

This sounds like a great addition to the release process. Thank you for putting time and effort into trying to meet the needs of developers! I hope your week-long meeting in Brighton helps solidify the new process.

@TJ: They are, but as with any development group, there are limits to their resources. Naturally, the bulk of their resources need to go to Unity itself.

Personally, I think this is a great first step. I’m well aware that major release process overhauls don’t just happen overnight, and do take a real concentrated effort, so I’m happy just to see things moving in that direction and hope to eventually see even more progress on that front.

If I had to choose…the Unity team spending time pulling apart Unity to have easier real patch updates vs adding and improving its game development features and abilities. I’d go with the latter.

Thank you Unity team for trying your best to get patch updates to us. Though one day…if theres somehow a time where Unity is no longer hounded for adding more game development features, that should be the time you put out a team to see if you can crack unity open.

As a developer who has worn multiple hats (maintenance, qa, support, beta, feature) for a medium-sized company (~500 TMW Systems), I can attest to the difficulties in building a patching system. Thank-you for your hard work in at least being able to get out fixes this way, and I look forward to the module manager. Few people will fully understand the difficulties and constraints of limited developer resources with so much to do.

@TJ — they are a development company to develop Unity, not patching software. When working with legacy code there are constraints and it can’t all be rewritten overnight to fit a patching system. Some things are more important, like getting 4.4, 4.5, and 5.0 out the door.

“We operate under the covers of our current technology, so we can’t just invent a new delivery system in this team” Aren’t you a development company? Isn’t that what you do, invent new software? What am I missing? You should have said, this is easier, so we don’t really want to.

@VINÍCIUS @NOVACK – 4.5 will ship with a feature called Module Manager which will allow platform code (iOS, Android, WP8 etc) to be updated without needing a full editor installer. It’s easier to use the existing build systems (as Thomas said) rather than spending our engineering effort building or qualifying a patch delivery system.

@Peter: The bugs we will fix in this team depends on a long variety of reasons. I tried to mention some of them.

@Vinicius: Not sure what you mean?

@Novack: We operate under the covers of our current technology, so we can’t just invent a new delivery system in this team. If you are not affected by these bugs, I suggest you wait till we roll them up to fully regression tested builds and released through the editor update.