Goodbye Beta, Hello Build!

Published April 8, 2020 by
Geoff Perlman

It has been tradition in software development for a new version of an app to go through a number of stages. Typically this starts with the Development stage. In this stage this new version of the app is so early that if there’s any testing done, it’s being done only by the programmers themselves. Next comes Alpha when the app is still not complete but far enough along for internal testing.

When the app reaches the point where it’s feature complete and ideally all the bugs that were planned to be fixed for this version have been fixed, the app moves to the Beta stage. Broader testing follows and may include users outside the organization that built the app. Often more bugs are found and some are fixed while others are left for another release.

When it is believed that the app is likely ready to ship, it reaches the Final Candidate stage. At this point there’s a build that is believed to be ready to ship. Those who are testing are then looking for anything that has somehow been overlooked and is severe enough to require fixing before shipping. This results in another Final Candidate. Once this stage is over, the app enters the final stage: Release.

The problem with this approach is that it does not result in uniform testing. Many users who genuinely plan to test early and test often end up waiting until a Final Candidate is available before they begin testing in earnest. This results in a barrage of reports at a point at which the team is preparing to ship, meaning either the ship date must be pushed back or bug fixes rescheduled for future releases.

Over the past several years more and more companies have been going away from these traditional stages. We have decided to make this change as well. Starting with 2020r1, the pre-release versions made available for testing will no longer have a stage attached to them. They will simply have a build number. We build almost every night and not every build is ready to be tested, so you will see gaps in the build numbers. This means the first several pre-release builds will not be feature complete. Once we are feature complete we will include installers. That means this will come much earlier than usual. That doesn’t mean we have stopped fixing bugs. That will almost certainly continue and new builds (with installers) will be released for testing. Once we feel the quality of the builds has reached the point where it has met our standard for release, we will ship. That means there will be no Final Candidate either.

We encourage you to test early and test often. We understand that not all of you can do that. But for those who can, we appreciate your effort and believe that this system will result in improved overall quality and a more reliable schedule for our software development.