December 18, 2004

The Definition of Beta

Several people who've visited this site the last few days have defined 'beta' in the traditional way, as something distinctly unfinished, full of bugs and therefore to be regarded as something potentially dangerous, while others see it on everything and think it's lost it's meaning. Last night a developer friend called, and I mentioned the posts and the term 'beta' and asked for clarification and opinion.

Since everything is now marked 'beta', the term has lost it's meaning to users, according to many commenters as well as most of the folks I've talked with, including several developers. As this software developer said to me last night on the phone, it's like having Jordashe slapped on the back of your jeans in 1978 at Studio 54.. it's trendy, it's hip, and everyone does it to be cool. But do users pays attention? According to this person, it's just marketing, fashion, and a way to not commit... and since so many have issues committing to things in our society, users just sort of think developers are having trouble committing to finishing.

Google puts it on everything but search, so does Flickr, so do many software sellers, as Brent did at Ranchero. It's clear now that he obviously meant it in one strictly definied way, but since there is so much variety of definition and use by other developers, how should users take this overall? Isn't it up to developers to address a common definition so as not to confuse things.

Until then, blaming users for misunderstanding the definition of 'beta' seems unrealistic, and while developers can make users 'bad' for doing this, developers actually are the leaders here. They have the control and power to name and define. Developers have confused this issue, and users should ask them to get this aspect of software clear in the marketplace. Blame Google, et al, for causing problems like this. For now, users have been trained by the aggregate of developers to think 'beta' doesn't really mean anything.

Charging users for 'beta' reinforces the loose definition of 'beta', because we assume the software is for real. When I purchased NetNewsWire, the Ranchero site let me buy it, and didn't say anything about reporting bugs or that it was buggy on the payment page, was to be strictly, traditionally defined as a 'beta' or that there would be a loss of data, which they were aware was a distinct possiblity upon putting in the license code. I'm sorry that NetNewsWire continues to be the example here, because lots of other developers have done the same thing, but my experience was with NNW and so it's the concrete example I have to work with for this write up.

Not charging for beta means users will have a significantly lower chance of confusing the definition of 'beta' as it's traditionally held by developers.

While most small companies would define Beta in the traditional way, it is the big guys' marketing departments that are diluting the meaning of the term in regards to software. If in doubt - blame marketing ;)

So Mary, as a developer and not a marketeer, and with you being the nearest user - how do I market my product in such a way that you *know* its not safe to use and without using the word Beta?

That seems clear to me. There's a nice, obvious warning on the site. If that weren't there, I'd say you had a point. What I'd like to know is, why weren't you using 1.08, which is clearly marked as a production version? I don't think you can blame Ranchero for the sins of others - they seem to be quite clear on the difference between beta and production...

I talked about this on my site but I want to address James' comment. Yes, there is a warning but I was easily able to bypass and go straight to the order page. I went to the front page and clicked the "Buy Now" button. I'm sorry but that beta notice should be on the order page as well so it is nice and clear, especially since it is extremely easy not to see it at all.

My other point is why should anyone pay for beta software? Why charge for it? In truth a "public beta" is a timed-trial, not purchase.

If a company/person charges a person money for a product it should work, within reason. There should definitely not be a "Beta software has bugs! Nasty, vicious bugs with great big, sharp teeth!" warning on it.

And Ross - the answer to your question? Don't charge for it. That shows a clear sign that it is beta - with no warranties. I'm pretty sure if Mary had seen a free "beta" version and the paid version, she would have picked the paid version for mission critical items and perhaps tested the beta version.

Hi Ross,
I'm not sure what the answer is, and this is partly why I ask the question, to get a range of responses. One thing might be to use the word 'test' instead of 'beta' because people who are not engineers usually have a clearer idea of what a 'test' is.

As I said earlier, a beta that gets charged for is implying something, to users. Maybe not to engineers, but users see it as just a word that gets used on everything, and the $$ charge confirms to users that it's an endless beta, with no distinct difference from the other software.

My recommendation would be to do user testing, see what catches 85% of users understanding, and go with that terminology.

Hi James,
If you read the threads in this and the previous two posts, you'll see that I said before that I was sent the software from a developer who recommended it (not someone at NNW) and maybe it was dumb to trust them, but I did it. Regardless, due to deep linking, it's possible to go buy NNW's stuff, and other sites allow deep linking as well, which means users can get to buying the software without seeing the warning.

However, even upon seeing that language, it's possible that a user could think it was left over from the past, because static sites are slow to update, meaning the warning applied to an older, less stable version from before they started charing or that it is a disclaimer because of a desire to avoid committment to finishing the software, or because the developer wanted to make a joke for a little fun. As stated in the post, the word 'beta' is not clear, has been overused, and isn't taken seriously by users and developers alike.

That's why I'm asking for constructive feedback about how to name test or beta software as something truly in the testing phase distinguishable from 'beta' as in Google News or Flickr or whatever.

There is a nice application review for an app called Delicious, the beta has a nice splash screen ... http://arstechnica.com/Media/2004/11/5/beta-splash.png

The full review if anyone is bored is at http://arstechnica.com/reviews/apps/delicious-library.ars
but the icon make me wonder whether the under constuction image was enough, or whether someone with a better grounding in semiotics might be able to come up with something iconic way of letting users know at the app level (in case they got it by mistake). If NNW had a similar splash Mary, might that have put you off?

Posted by: Ross at December 18, 2004 02:53 PM

its obvious deep linking to downloads where the download is beta but no warming that it is beta on the download page ought to be prohibited because of click happy users whose brains are so multi-task scattered they are not really conscious of what they are doing or aware of relevance like in misison critical and leave deep linking to the rocket scientist who would backtrack to the home page to see if there is any plain english.

Posted by: baxter case at December 18, 2004 04:33 PM

Mary: You're buying into fashionable complaints about the "beta" designation, but that's all it is... fashion.

Flickr and similar companies aren't calling their wares "beta" as a marketing gimmick... they're using the term because it sums up the essence of their situations. "We're doing our best with our resources, but the future is uncertain. You can use it, but don't count on anything."

After all, there are bugs in places other than code. In business models, for example... a photo-sharing service may never turn a real profit, they may have to pull the plug unexpectedly, and every photo on the site could fall into a black hole. Or they could get bought out, and the new ownership could radically change the rules of engagement for users. Or it might prove so ridiculously successful that they have scaling issues, and end up minimizing or under-delivering existing or promised features. Or the engineers may realize they've hosed something up in the basic design, and end up doing a rewrite that isn't backward compatible with what was there before.

People need to be warned about that kind of possibility, even if things seem quite solid on the surface. Perhaps *especially* when things seem solid.

This even applies to something as stable as Google News. The bottom line is, Google still doesn't know how to monetize the thing, and that's a riddle they may never solve... in which case, Google News could just disappear one day after a contentious budget meeting. If they weren't warning you about the unfinished, impermanent nature of the thing, they would feel obligated to either stop people from accessing an otherwise useful tool, or leave it running past the point where they can reasonably support it.

As for paying for beta apps... when it comes to a small developer, it's often a case of "either you pay for it, or it ceases to exist." If you don't have venture capitalists paying your credit card bills, supporting users costs money... unless there's an influx of cash from *somewhere*, there's no product there for anyone to complain about.

To sum up, the meaning of "beta" is already pretty much universal... "not for mission-critical use". Don't bet your career, your project, or your life on a beta product. Simple as that.

Basically, whether the google links you clicked say so or not, it *does* still mean beta to the software industry. It's a technical industry term. Alpha, Beta and Golden Master are part of the production cycle taught in CSc classes around the country (or they were last time I bought a text book anyway!).

At worst, perhaps programmers are guilty of not making the definition of beta and the status of a beta app more explicit, as mentioned in the Delicious example above. A really big BETA stamp over the startup screen is a start. And maybe a dialog box with a definition of beta every time you install a new beta version of beta software is also a good idea.

But I'm basically in the camp that says that you used beta, you found a bug. This is how beta works. I encourage you to be more disciplined ritual for saving your work if this really upsets you that much. In fact, I encourage everyone to be more disciplined in backing up their work. It's not like programs that don't have a beta sticker on them couldn't lose your work too. For that matter, it's not like an unexpected power surge, a leak in the roof above your computer, or the cat gnawing on the wrong cord couldn't all have the same affect. Nobody intentionally did anything bad to anyone else here. Computers lose stuff sometimes. End of story.

I have a really great sign on the wall in my office that says "Back up your data, the end is near!" I think that sums up the state of every single computer in existence, speaking in very broad terms, of course...

Posted by: heavyboots at December 18, 2004 11:15 PM

This is a great topic. I had mentioned on a beta site that I've been playing at that there seems to be a corruption of what beta used to mean. With the mention here of "traditional" beta, I see more about what the pattern is.

First, what is refered to here as "traditional beta" isn't. But it's worse than that.

The first use of widespread beta as a form of market seeding and early-adoption, as well as market feedback, is a collapse of several different activities and purposes into a mess. It seems to have arisen with MSFT and their becoming proud of the number of "beta" shipments that are done before reaching release-candidate position for major products. Although there may be several missteps up until the fatal one, the fatality was confirmed the first time a beta was shipped that (1) did not register all of its users specifically as participants in the beta test, (2) did not extract an NDA or other agreement about the purposes and limitations of the beta test and (3) did not expire to be replaced, under pre-agreed terms, with the production release.

The use of "beta" on a web-based service is the ultimate extreme of this practice. First, there is no notion of who the users are, there is no special feedback mechanism for reporting beta-user experience, and there is no agreement about what might or might not happen in moving from beta to production/stable or whatever. There is also no configuration stability for even tracking user reports against builds and the version of the system that was running at the time an user had an experience they have gone to the trouble to report. (I am overgeneralizing, some beta sites might provide that. I'd like to hear about it.)

So the whole thing has been corrupted and Mary's right, the techies own the nomenclature (and appear to have ruined it, as far as I can tell). But the tradition, in this case, is one of the corruption of language, apparently through mercantile puffery.

During my 6 years of software engineering in the UK, and first 5 years in the USA I never came across a use of "beta" that had anything like the connotations it does nowadays. After entering the dot com zone of software all that changed. My first employer in the USA had probably the best beta program. A product released for beta had to be feature complete, had to have gone through the same testing as the final release would, could not have more than a well defined set of bugs in it (i.e. no critical bugs), and above all had to have an identified set of customers who were willing to devote time to using the product and providing feedback on it. The beta programme was typically six weeks and all customers had to provide written feedback and work with a beta programme coordinator. Engineers would be required to provide direct support to beta customers as QA would not be fully trained to support it yet.

After beta a product would go through and additional development cycle to fix any bugs that cropped up but NOT add features. After beta the next release would be the final aka FCS (first customer ship). Occasionally important customers would get a rush release between beta and FCS that could have incomplete docs, some bugs not fixed yet etc. etc. this was know as a pre-release. Any kind of release before the beta that was not feature complete was called an alpha release. However even an alpha was generally accepted to have a well defined set of features, of which some may not be complete - new features would not be added for customers after an alpha without considerable thought and planning.

So, in my mind, I still live with this product life cycle nirvana and wince when I see current loosey goosey definitions of "beta" where beta is ipso-facto assumed to be feature incomplete and full-o-bugs. I believe that software should be released early and often, but not at the expense of releasing untested buggy software. Complete the features, test them and release with lots of 1.x type point releases along the way as new features are added. Do 1.x.y releases for maintenance bug fixing only, but don't add features. Maintain customer compatibility without any pain for all minor releases. Provide migration tools for major releases and pay significant attention to planning such migrations if necessary with your customer base. Unexpected data loss is never acceptable.

The people who told you that there is widespread confusion over the meaning of beta are simply misinformed.

The developers have NOT confused things, and nobody is confused. A test release is a test release.

Google betas are betas; they might change or vanish.

A similar "controversy" emerged recently over open rehearsals at the Boston Symphony Orchestra. Often, these rehearsals are essentially run-throughs -- a lot like concerts. Some people, accustomed to these run-throughs, are upset when a conductor stops to correct a problem.

Beta or not, software is imperfect. It doesn't matter what you call it.

Software users have been conditioned by the software industry to expect perfection but this is simply not realistic. It was only possible in the past to get very close to defect free because the problems were smaller, and so was the software.

In the present day most software is so large and complex that it's just not possible to remove the majority of the defects, even known defects. The product would never get out the door and the business would fail.

Many software companies are making free updates the new status quo in software. Some of them release "beta quality" software as 1.0 and don't get it right until 1.5. The term "beta", just like version numbers, is completely arbitrary.

The good news is that users are expecting defects and expecting to update their software more often. Some are also expecting severe data loss, like you experienced. Severe data loss will happen more often, not less, as software gets more complex and is forced out the door by the eager business end.

The software industry needs to educate people better about the expecations they should have about the quality of their software (beta or not), and the consequences of failures. Then people may back up their critical data more often.

Hi Mark,
Thanks for the comment (also saw your blog post). So I'm wondering, if a test release is a test release, then why not make that absolutley clear by not charging for it?

When a developer shuts off access to the program after the 30 day trial, and a payment is requested and the user directed to a payment page, isn't that prompting users to see the product as relatively finished with testing and ready for charge / prime time?

That is the distinction I made. I was rather surprised to get the charge request, but assumed this was why they were making the request for payment. Because beta = test / free.... charge software = bugs pretty worked out and ready for prime time.

So the question for you is, when do you make the distinction between beta/free and charge software? When and under what circumstances do you move categories?

Also, even though you as a develoepr don't feel confused about what the terms mean, other users may, and have, as other commenters have mentioned. So as a developer, would you be willing to lead something to correct this problem that users have because not all developers are as clear as you are, to keep the use of 'beta' or 'test release' terminology clear and distinct, from 'relatively finished' or GM software that comes with a fee?

This is ridiculous. You picked on the wrong piece of software and the wrong developer. The software is beta. It says it's beta. If you spent 5 seconds on the distributor's web site (of a product supposedly indispensible to your livelihood) you would see a) it's beta, b) there's a prominent warning and c) there's a 1.0.8 NON-BETA version!!

While the fact that there is a charge for the beta is totally irrelevant to the whole discussion (if you don't want to pay, then don't pay!!!), did you ever consider that perhaps part of the beta was to test the payment process and unlocking the software??

Trying to make the argumet that the term "beta" has changed its meaning is a complete cop out.

Posted by: pb at December 19, 2004 09:46 PM

Mary, it's like you were listening in on a conversation we were having last week. great post. here's what we posted on our blog... The Bastardization of Beta
http://www.hypergene.net/blog/weblog.php?id=P237

I read the HyperGene comment Shayne points to and which is more consistent with my experience (along with BlogGently's observations). I am bothered by Mark's comment and his treatment at http://markbernstein.org/Dec0401/OfTemperandPrice.html. The part that is weird for me here is the loose language (calling a software failure a "mistake") and the paternalism (telling readers there that there is nothing to be learned here and explicitly not posting a link to the original discussion). Finally, in the comment here, Mark is appearing to tell people who claim to be confused by the use of beta that they aren't. I don't remember when I learned to stop telling people how they feel or what their experience is (or should be), but when I encounter anything like that now all sorts of alarm bells go off. I think this is a good reason for listening to Dave Winer when he says "the customer is always right." As a practice, it can take us out of that destructive, listening-only-to-ourselves posturing.

We have had available to us, for years now, software engineering techniques designed for rapid development and frequent release while keeping bugs under control. Most of the industry doesn't use these techniques, and as we can see from the posts here, it seems that even people with software engineering degrees aren't aware that they exist. It's prefectly possible to do a new production (non-beta!) release every week of a major software system, and make sure that there's not going to be big nasty bugs in it; there's really no excuse for data loss except that you decided, as a product manager, that you wanted to ship a product with a risk of it.

The tools are there; the knowledge is there; programmers and managers in general are just not smart enough or not driven enough to take advantage of them.

Posted by: Curt Sampson at February 14, 2005 03:22 PM

I heard recently that the word BETA meant death...but I can find no reference to this. Was that misinformation? Yhank you. J