Wednesday, June 30, 2004

Paranoia == Good

This will sound a little bizarre, but one of the proudest moments in my career as a consultant to Fortune 100 companies came as the result of a major system meltdown. And it highlights why my "paranoia on steroids" mindset is a pretty valuable trait in the software world.

As a bit of background, I need to explain how a large Lotus Notes shop works, at least as it pertains to email [1]. Notes, among other things, can serve as the backbone of a large, distributed email infrastructure. In this particular IT shop, Notes was distributed globally (not an uncommon occurrence) over quite a large number of servers.

The databases within Notes undergo a task known as "replication" or, more specifically, "multi-master replication". This means that changes to any server are merged with other changes from related servers on a near real-time basis. This is one of the advantages of a distributed Notes systems: changes can occur anywhere in the system and they will automatically ripple through the entire network without user intervention, so everyone is always up-to-date. Occasionally, the same database record will be modified on two servers at once - this situation is known as a "replication conflict" and Notes will automatically remember both copies of the record for you.

One of the key databases held in each Notes server is a list of user accounts, known as the NAB. The NAB stands for "Name & Address Book". The NAB is pretty important because it contains all valid Notes user accounts on a given system and all of their personal information.

One of my areas of expertise is directory servers (LDAP directories) and, as a consultant, I did a fair amount of architecture, design and deployment of directory services. In this shop, among other things, I created a system-level process that would take a feed from Notes (since, at that time, it was the authoritative source for internal user accounts).

The way my feed process worked, high-level, was something like this:

> Read all of the Directory's user accounts into a map

> Read all of Notes' user accounts into another map

> Rip through all of the Notes accounts, comparing each one to a Directory account. If they compare, remove the account from the Directory's map. If they don't compare, save the changed fields into an Updates list.

> When done, save the remaining Notes accounts into an Inserts list.

> Save the remaining Directory accounts into a Deletes list.

Okay, that all seems pretty straightforward. But here's where my paranoia kicked in. Notice that instead of just doing the changes directly in the directory... I saved them into separate lists. Why?

Failsafe

Instead of doing the writes, our superhero, Paranoid-Boy, placed the following checks at this stage of the code:

> Is the number of Deletes greater than x% of the total number of directory accounts?

> Is the number of Updates greater than y% of the total number of directory accounts?

> Is the number of Inserts greater than z% of the total number of directory accounts?

If any of these values were exceeded, I assumed that either a catastrophic fault had occurred or that manual intervention for some major change in the user population was necessary. If this happened, the feed would reject the entire operation, notifying the operator and giving them the option to perform a "manual override".

That's why I didn't do the writes directly. I wanted to wait until I could 'sanity-check' the number of changes to the directory, which was the key repository of security data.

Sure enough, the alarm claxons went off one day. Apparently, a bad feed process had corrupted one of the central NABs and removed 12,000 users. Notes, doing as instructed, replicated the change throughout the planet before the system administrators could get a hold of the problem. So in (probably every) NAB around the world, all 12,000 users got purged. And I think some pretty senior management folks (maybe even the CEO) temporarily lost their access to Notes services. That's a tad embarrassing for the IT execs.

Worse still, the systems downstream were at huge risk, including the Directory. Luckily, my feed process' paranoia check kicked in and rejected the feed from Notes: too many changes! No one lost their security privileges due to the Directory and the folks on our team were safe in their beds that night, sleeping like babies... while the Notes sys-admins were, I'm sure, pulling all-nighters trying to get it all back together.

A little long-winded, I know, but my point is simple. None of this would have happened if the rogue feed process had some paranoia designed in from the start.

If your system is non-trivial, be paranoid. Be very paranoid.

[1] This information is somewhat dated, so don't expect you can get your Lotus Certification after reading this junk. In fact, don't even try to use Notes based upon anything I've ever written or said. Please use professionals like Pete for all things Lotus.

Tuesday, June 29, 2004

Software Poseurs

I noticed an interesting thread running on the JOS forum. One of the original posters wrote:

I am not pleased with these Express Products.

Take a look at the blurb on the Express page. They're targeting "enthusiasts" and "hobbyists". There is no way that this is a good thing. Remember when Joel said:

"There’s something weird about software development, some mystical quality, that makes all kinds of people think they know how to do it. I’ve worked at dotcom-type companies full of liberal arts majors with no software experience or training who nevertheless were convinced that they knew how to manage software teams and design user interfaces. This is weird, because nobody thinks they know how to remove a burst appendix, or rebuild a car engine, unless they actually know how to do it, but for some reason there are all these people floating around who think they know everything there is to know about software development."

Well, these Express editions are just going to foster that type of "there's nothing to software development - anyone can do it!" type of thinking.

I say again...there is no WAY that this is a good thing. It's just going to further commoditize development talent.

Of course, this set off a Slashdot-esque cornucopia of responses ranging from, "What a load of elitist crap" to "I've made a reasonable amount of money being called in to fix disastrous 'applications' built by hobbyists who were able to sell themselves to small business owners who didn't know any better."

My take: is this anything new?. Lowering the bar for poseurs in the computer field reached its zenith in the mid- to late-nineties. At that time, the twin juggernauts of the world-wide web and Y2K had pitched even the most conservative companies and governments into an IT spending boom the likes of which we'll never see again [1].

For the most part, the downturn that began in April, 2000 flushed most of the poseurs down the swirling vortex of history. Projects that relied on a Liberal Arts major who had taught himself HTML, an MCSE with six months of real-world experience, and a graphic artist ("to make the site look really sweet") ended up in the dustbin of expired domain names.

In other words, most companies have come to their senses. This isn't rocket science, folks. You always get what you pay for.

The lesson learned still applies today. Companies that award projects based solely upon rate have -- and will continue -- to get burned. I know one such CIO who has scorch marks on his rear from repeatedly late, restarted and altogether failed projects. The smart senior managers find talent in every venue. That talent can come from India, China, West Virginia or Boston.

So if the act of turning data into knowledge relates to your business, you will need to have better bit-wranglers than the next guy. And they're not too hard to find. They're the guys with the stellar track records who can't be brought in on the cheap. Supply and demand doesn't change, even if the number of poseurs is an order of magnitude higher than what it is today.

[1] While I hate to mix politics with technology :-), I will go so far as to say this. The majority of Bill Clinton's "economic expansion legacy" was due to two simple twists of fate:

1) Tim Berners-Lee, who just happened to invent the world-wide web, and:
2) The Y2K frenzy (everyone who still has Krugerrands and over 1,000 rounds of ammo in their basements, please raise your hands)

The fact that both overlapped with his Presidency is something he should give thanks for every single day. That being said, the downturn that occurred in April, 2000 is no more his fault than Y2K.

Monday, June 28, 2004

The Alpha Email Collection, Part 705

Email #1: Guinea Pig Policy

After Alpha posted a lengthy Dog policy (I believe Fridays you could bring your dog into work, subject to a plethora of rules), Gerry sent me and Dave this email.

Alpha has decided to adopt the same Guinea Pig policy that several other liberal software companies in the area have developed. This policy is as follows:

1) Live Guinea Pigs must be immersed in a mixture of water, pet shampoo and Lubriderm for 10 minutes before entering the building. This cuts down on the dander typically released by the animal, gives it a shiny coast, kills fleas, and usually the animal itself. They must be on a leash and are not allowed in the weight room or in the reception area.

2) Recently dead guinea pigs can be brought in after laundering them in a washing machine through a wash, rinse and spin cycle. The remains may be brought on a leash only. Drag marks must be cleaned up by the owner.

3) Decomposing guineas pigs may be brought in only when wrapped in wax paper, sealed in a ziplock bag and frozen overnight. The animal must remain in the freezer at Alpha all day except when the owner takes it out for a walk (or lunch). Frozen remains left on one's desk will be discarded.

4) Stuffed guinea pigs may be placed on the owner's desk as long as the animal has been properly dried, treated and stuffed. These make good paperweights, pencil holders or 'Nerf' [tm] style footballs.

5) On the second Friday of July, Alpha will sponsor the annual "Guinea Pig Fandango". Prizes for biggest, fattest and ugliest will be awarded along with the "Stupid Guinea Pig Tricks" and Guinea Pig Toss competitions.

6) While this policy embraces our furry rodent friends, animals that step out of line will be tossed in the disposal in the lunchroom.

Thank you for your attention.

Email #2: Shiftless Frivolity

Let me paint the picture for you: our Marketing Department took some photos, taken inside Alpha, with some of the better looking employees as models. I can't remember the purpose -- perhaps there wasn't a good one -- but one photo in particular was perfect for a "fill in your own caption" contest.

It showed Selwyn standing over a conference room table, presenting to Bob H and Karen, positioned as if they could even fathom what he was talking about. Just kidding. Anyhow, Bob solicited dialogue from everyone in the company and sent out a "best of" email shortly thereafter.

Some of the ones I found entertaining:

From Terry

What Bob and Karen hear from Selwyn: "We must start working toward blah blah blah blah blah blah blah blah. What do you think, Karen?

Bob: [to himself] "Whew, I'm glad he asked her."

Karen: "I think Bob could answer that better."

Bob: "...I think we should ask Richard."

Selwyn: "You're both fired. I should have hired more development type people.

From Laura

Selwyn: Karen, why is it that Bob has nothing better to do than send useless emails to everyone in the company? And who dresses him?

Karen: Didn't you hear there was a big scandal at the Garanimals factory? Massive amounts of tag switching were done, apparently at random. Bob is living proof that white collar crime is not a victimless crime.

Bob: Hey, leave me alone! My butt hurts and I've got emails to send!

From Pete

Selwyn: (Interpret the smile as an evil sarcastic grin) "I've had it with marketing's shiftlessness! If you don't get back to work now, I am going to rip off your heads and feed them to development!"

Sunday, June 27, 2004

Spyware and adware... reloaded

Had some (ahem) fun today removing spyware from my kids' wireless laptop. My kids, despite their knowledge and best intentions, have inadvertently polluted their machines with some of the most virulent adware/spyware trojans known to man.

So with two firewalls, virus scanning, AdAware, and SpyBot installed, how do these kids do it? Usually, it's as simple as visiting a nefarious web site with Internet Explorer. The kids know not to accept any downloads, e.g., the ubiquitous "Would you like to download the Super-Mega-Bargain Free Stuff and Cash Back Toolbar with Universal Time-Clock Synchronizer (and a mess o' evil sh*t that will f**k your machine up) application?".

The main approach for getting kids to these sites is through instant messaging aimed (no pun intended) at pre-teens. The black-hat will create a pseudo-funny/entertaining, but entirely malicious, web site and get IM conversations going, either manually or automated, advertising the URL. Soon the URL is being passed among the innocents, and before you know it 40,000,000 machines have been turned into zombies for DDOS attacks or porn-spammers.

The strange thing is, Internet Explorer contributes to this disastrous mess in two completely different ways:

1) It has tons o' nasty exploits (that make it impossible for anyone to keep up); worse yet, some of the patches don't work right or can't be applied. What happens then? Throw the machine into the trash compactor? IE and its components go back to the days of "Writing Solid Code", one of the single biggest drivers of sh*tty software ever. I'll have a full rant on that book when I get some time to outline its egregious mistakes.

2) It supports a (toolbar) extension system that is the first victim of an aggressive spyware/adware vendor. Offhand, I couldn't tell you how to clean up hidden toolbars that have hooked into the underlying HTTP networking layer (called WinInet). But I do know that it's rife for exploitation. Why can't IE warn the user that an extension has requested installation? Oh, that's right, they don't give a crap.

Anyhow, back to the removal process. When AdAware and SpyBot don't get the trick done, I rely on the tried and true trick of checking out Task Manager to see what's happening. Hmmm... don't like the look of that process: jm39yg12.exe. I kill it. Cool. Uh oh, another process pops up: yz42ely3.exe. You can kill these things off to your heart's content... and they keep coming back.

Okay, I'll hit the registry and find out what's kicking these processes off. The registry editor is a manual configuration tool that lets savvy users change low-level settings. Not for the faint of heart, but the basic "automatic run at startup time" entries are pretty safe. Whoaa... the malware shuts down my machine when I try to run the registry editor.

Okay, restart. Command-line. Copy regedit.exe to a different name. Now I can edit the registry. I find all the favorite places where these *sswipes like to save their stubs and kill them off. One of them is called automove. It's probably the process that keeps spawning off new copies of randomly named tasks that are up to no good. Here's the trick: don't remove the registry entries... just modify the data to point at non-existent processes. In some cases, removing the registry entry doesn't accomplish anything. Some of the checking processes will recreate the registry entries. But they apparently just look to see that the keys are there... but don't validate the data inside the keys.

Anyhow, I have the machine somewhat cleaned up. And whoever wrote the "automove" app: I will hire you. You have a standing job offer.

How can we, the innocent consumer, fight back, even if we're not technically savvy? Easy. Every popup ad you see, note the domain name of the advertiser. I mean make a list of ".com" domains. Create a complaint letter and send it to:

with your complaint (no threats or viciousness - try something worse: you'll never give them a cent nor will anyone else with whom you're familiar... because you'll be spreading the word that their marketing practices suck).

Democratic disorganizer: helped fragment the left by convincing failed candidate General Wesley Clark to run against an already weak, crowded field.

Free-trade Outsourcer: believes in outsourcing American jobs to other countries, having run his own business in outsourced fashion.

Anti-Richard Clarke: has repudiated Richard Clarke's contention that Clarke, not Bush, approved the Bin Laden family's departure from the U.S. following 9/11.

Anti-Franken: has distracted America's limited attention-span for left-leaning pundits from Al Franken's monstrously unsuccessful Air America Radio, down to 15 markets world-wide. Moore's film might actually be a death blow for Franken's critically ill venture.

Galvanized the left against egregious anti-Bush rhetoric: even the left-leaning Slate has ralled against Moore's movie: "...To describe this film as dishonest and demagogic would almost be to promote those terms to the level of respectability..." - this from Christopher Hitchens, a columnist for Vanity Fair (of all places).

In fact, I wouldn't be surprised if Moore is actually 6 feet tall, 180 pounds, lean and mean, with a sculpted, masculine figure... who only goes out in public in a fat- and ugly-suit to help galvanize support for W. [2]

Thursday, June 24, 2004

DC...

In D.C. on business. Coincidentally, my sister-in-law is flying here today because my brother has been offered a promotion to run his company's Washington office. My brother -- an ultra-bright, ultra-motivated person -- is currently general counsel for an RBOC, running a state-wide operation. The promotion would, I'm guessing, involve running all regulatory details for the entire multi-state operation.

Anyhow, his wife is flying up to look at houses. They live in a large, metro area but not nearly as large, metropolitan or as expensive as DC. They have a big, beautiful house in a relatively idyllic suburban setting. But, from what I can tell, a house near or inside the beltway that resembles their current one could run well over seven figures. Holy shnikes! And there's even sticker-shock on private schools: tuition in the DC area is about 2.5 times their current cost.

I'm sure the reason my brother got the offer, though, is because he's about the best you could ever bring to bear. Have an incredible level of detail to master (say, tens of thousands of pages of arcane regulatory details)? Need an ultra-competent courtroom demeanor and split-second verbal skills? You can't get much better than him.

There was a good story about him in their local business paper a while back.

On a blistering September afternoon last year, ... Ross listened to an endless stream of ... companies explain why [his company] should not be allowed to enter the market for long-distance telephone service. [They were] not in compliance with FCC standards, they said. [They] had not opened itself up to competition, they said. [They were] not allowing other carriers access to its network, they said.
When it was his turn to speak in front of the ... Public Service Commission..., Ross held up a newspaper ad from rival MCI Corp.

The ad's headline read, "You now have a choice." With that simple maneuver, Ross virtually erased the previous two hours of testimony and helped ensure a favorable ruling for [his company]. "For the entire two hours, everyone was saying there was no choice for consumers...," Ross said. "All we did was show the commission that there clearly was a choice. And the commission agreed with us."

Because in addition to being smarter and more motivated than about anyone you could put up against him, he brings one additional card to the table. Ultra-competitiveness. He hates to lose. And I don't recall him losing too often, whether it was running for president of seven organizations his senior year in high school (winning all seven), making editor of law review at a very serious law school, and making partner in his first job in near record time.

Wednesday, June 23, 2004

P-p-p-partnering with Microsoft

Thanks to a John Lim link, we've discovered further evidence (as if twenty years of consistent behavior wasn't enough) of the folly of a small company "partnering" with Microsoft. And that's even if the market segment is in the critical 'developer tools' neighborhood.

Eric Sink is one of the founders of SourceGear Vault, a successful version-control system geared towards large teams and collaborative development. Now that the market for this type of product looks appealing, Microsoft has decided to enter it, announcing its forthcoming Team Foundation Server (TFS). Now that's their right.

But you ask yourself: why would a small development house, one with no leverage, choose to partner with Microsoft in the first place? After all, their expectation at some point must have been -- if the market proved big enough -- that MSFT would enter the market full-force once it was established in a meaningful way.

And I'll leave the answer to Eric. But my guess is that any tool, which needed tight integration with Visual Studio (the Microsoft development environment), would also require 'special' documentation on Studio. In other words, the really detailed doc that describes all of the low-level glue necessary to make integration truly work. So I wonder whether the price of admission -- to get your hands on that detailed information -- was participation in a partnership program.

And there's the dilemma for an ISV. In his blog, Eric describes the scene when he found out about TFS:

The scene of this announcement was weird. It was an NDA'd meeting of the VSIP program. All of the attendees were vendors of components or developers tools which integrate with Visual Studio. Standing in front of dozens of their so-called "partners", Microsoft staff put smiles on their faces and announced that they will be competing with almost all of us.

There was a lot of emotion in that room. Every VSIP vendor had worked very hard to be a part of Microsoft's "ecosystem" for Visual Studio, only to end up feeling very betrayed. If this is how Microsoft treats its partners, how do they treat their enemies?

So, my recommendation for small software development houses: if you can avoid it, don't partner with Microsoft. Accept a buy-out for a reasonable sum, but don't partner. Integrate (or at least have the option of integrating) with competing, multi-platform technologies wherever possible: MySQL, PHP, OpenOffice.

I love Microsoft's development environment and many of their products. In most cases, they are simply the best you can get. But any small development shop better have their eyes wide open.

Tuesday, June 22, 2004

Memorial Day - a Novel

I just finished Vince Flynn's new thriller Memorial Day and posted this review on Amazon:

At last. High-level CIA operative Mitch Rapp is back. A Rapp-led special forces team wreaks havoc on a terrorist enclave located in no-man's land on the Pakistani/Afghani border. What they find in a terrorist bunker is literally earth-shattering: a map of Washington DC complete with concentric blast circles. And it's apparently a damage estimate based upon the planned detonation of a 20-kiloton nuclear weapon near the DC downtown.

Through violent, but off-the-record means, Rapp interviews the five remaining terrorists. Only a couple survive the interviews, but Rapp has most of the information he needs. A nuke is scheduled to enter the U.S. on a container ship from Karachi. While the weapon is being tracked down, an ironic turn of events is occurring within the Attorney General's office. The Patriot Act is a political hot potato and senior members of the State Department are moving to kill the Act to save the president's re-election bid.

Rapp, true to form and ignoring politics completely, is more concerned with another issue: is another nuke entering the country as a fallback plan for the terrorists? And when the State Department starts treating the nuclear threat inside the US like it's a law enforcement problem, Rapp unleashes the dogs of war.

After Flynn's last book -- a disappointing effort in my view -- I made a few suggestions. Amazingly, he seemed to take them or, at least, arrive at them on his own.

1) Anna Rielly out of the picture: check
2) The real war on terror is now front and center: checkety check
3) Rapp goes on a Creasy*-like revenge extravaganza: check and mate, beeotch

Memorial Day is more than a pot-boiler thriller; it raises the compelling issue of human rights. At what point, for instance, is it acceptable to torture a suspect when the information could save a hundred thousand lives? I rate this novel 9 out of 10. On the merits of his latest effort, Flynn is back in the pantheon (Raymond Chandler, Lee Child, Nelson DeMille, CS Forester, and *AJ Quinnell)... on probation, but moving up fast.

Sunday, June 20, 2004

Letter to the editor of the Cincinnati Enquirer, which I'm sure won't be published. Too many other folks have responded to various and sundry naive folks decrying the Iraq war because 'there was no direct tie to Al Qaeda'.

To the Editors:

Those who demand evidence of a direct link between Saddam Hussein's Iraq and Al Qaeda are missing the point. Before our armed forces paved the way for a free Iraq, Hussein's government:

- Harbored the "most wanted" terrorists from the 80's and 90's: Abu Nidal and Abu Abbas (Nidal led the '85 hijacking of the Achille Lauro)

- Provided training camps for terrorists, including a Boeing 707 airliner at Salman Pak used for the training of hijackers

- Made cash payments to the families of suicide bombers

- Provided official support for PLO, Hamas, and Islamic Jihad; and harbored the Al Qaeda affiliate Ansar al-Islam

Thus, Hussein's large-scale human rights violations, support of terrorism world-wide, and development and use of WMD's made him a first-priority target in the war on terror.

Those who advocate appeasement, or rely upon conflicted organizations like the U.N. to protect us, need only read their history books. Inarguably, tens of millions of lives would have been spared had Neville Chamberlain forced a direct confrontation with Hitler.

Now, with WMD's in the mix, the stakes are even higher. The risks of prosecuting a less aggressive war on terror are too high to calculate. Trying to appease terrorists, or hoping that conflicted international organizations will shield us, is a foolhardy and potentially disastrous approach.

Saturday, June 19, 2004

Putting Excel Spreadsheets on the Web

I guess there's a problem with spreadsheets. Information gets locked into a spreadsheet on a desktop and there's no easy way to disseminate or share the data. Yet you'd be amazed how many critical business processes -- at some surprisingly large companies -- depend upon a lowly spreadsheets maintained on a user's desktop.

For many companies, replacing a spreadsheet with a real web application is a daunting problem: first of all, users are conceptually familiar with the spreadsheet paradigm and don't like to change. Secondly, it's tough to rustle up enough budget to replace ubiquitous spreadsheets when other pressing concerns are out there (say, hackers crawling through the firewall or that big ERP conversion that's taking four times as long as estimated).

The BadBlue product is designed to take most of the nastiness and complexity out of the picture. It comes complete with its own web server, hardened over years of hacker-testing. And, if necessary, it can run on Microsoft's IIS web server, with Apache support on the way.

You mark a spreadsheet as shared, defined some users, password-protect the file, and you're ready to go.

In less than a couple of minutes, you should have a spreadsheet ready to be shared on your LAN or, if you've installed it on an externally accessible system (e.g., via a router), you can have your colleagues reviewing it and updating it.

Here's the cool part: you can restrict what users can see and do. Don't want a clerk to see that sensitive "true cost" column? Read-protect it for that user - they'll never know the difference. Want your accountant to be able to comment on certain items, but not update any of your figures? You've got it. With audit logging and multi-user conflict resolution built-in, there's a fair amount of security tie-in that can help with Sarbanes-Oxley and other compliance processes that cause baldness in IT managers.

The major new stuff that's been added: a "find" feature (e.g., go to the bottom-most row so you can start doing data-entry... or find that company by name at row 7000). Also, an "insert row" feature to start pushing in additional data without messing your summaries at the bottom.

In addition to that, we'll be supporting a hosting option for spreadsheets, where for a reasonable fee you can have your spreadsheet hosted, multiple users set up to update it, and so on. Email me if you'd like more info on this.

Anyhow, if you're living with spreadsheets, check it out. It might save you some time and energy... and let you deal with that ERP implementation that's got you chewing Minoxidil tablets like they're M&M's.

Friday, June 18, 2004

Return-codes vs. Exceptions, part 516

Let's do a little recap and summarize what I've covered so far regarding the never-ending debate between return-codes and exceptions.

Late last year, Joel set off a veritable firestorm of controversy by stating that, in short, exceptions were as hazardous to a developer as 'goto' statements. His reasoning:

- They are invisible in the source code. Looking at a block of code, including functions which may or may not throw exceptions, there is no way to see which exceptions might be thrown and from where. This means that even careful code inspection doesn't reveal potential bugs.

- They create too many possible exit points for a function. To write correct code, you really have to think about every possible code path through your function. Every time you call a function that can raise an exception and don't catch it on the spot, you create opportunities for surprise bugs caused by functions that terminated abruptly, leaving data in an inconsistent state, or other code paths that you didn't think about.

The Java and C# community erupted with dissent, including Brad, Jesse, Sergio, Ned, among others.

Over the past few weeks, I've spent quite a bit of time describing why I believe return-codes should be used for return-code handling (rather than brute-forcing exceptions into a return-code model). In short:

1) Exceptions tacitly encourage abdication of cleanup responsibility. Nearly every example I've seen highlights the "readability" of exceptions and they often use strained, hardened examples of try, catch and finally blocks that work well when addressing the return-code issue. But the problem is, very few people are disciplined enough to set try, catch and finally clauses for every single method. Look at some real, production code. Odds are high that you'll find few methods that are covered for try, catch and finally. And, everyone will agree (I'm sure) that in those cases, exceptions are no panacea for unwinding state. On my development teams (C++/C), I've counseled developers that they should do as Joel does. Don't throw any exceptions and catch everything. All methods and calls return codes. Logging and instrumentation must be performed on any high-level routines. Example below.

2) Exceptions are significantly more difficult to maintain. Review some source code that uses exceptions. Real source code. Not the strained, artificial-looking examples of perfect exception handling. Look at the method calls. And tell me which exceptions are going to get thrown by which methods (without relying on a cheat sheet) so that you can cleanly determine where you'll set state variables to recover from partial completion. Return-codes encourage the use of state variables for unwinding state. It's crystal clear in our architecture that all method calls should operate the same way. They return error-codes. And every single one needs to be checked.

3) Exceptions complicate tunable logging and instrumentation, a valuable -- but often omitted -- capability of reliable systems. I haven't seen an exception-based model that tackles my logging and instrumentation example, below. I'm still waiting. For a more details explanation of tunable logging, see this previous blog entry.

4) Exceptions add a needless abstration layer over the return-code based systems on which we rely. RPC/LPC systems (e.g., SOAP). Operating system calls. Device-driver interaction. Guess what, when we retrieve a SOAP document, we get a status code returned within it that tells us whether a failure occurred and, if so, the type of error. Thus, the baseline for interprocess communication is... return-codes. The baseline for operating system communication is... return-codes. And the baseline for device-driver software is... return-codes. Do we need an exception model pasted on top of the native return-code model? Actually, do we need lots of arbitrary exception models (given the fact that C++, C#, Java and other languages diverge significantly on actual exception implementation - types of exceptions handled in various subsystems and assemblies, checked vs. unchecked, etc.)? Do we need a bunch of incompatible exception models floating around?

5) Exceptions are, in the vast majority of eligible systems, not used (and even banned, in some cases) for mission-critical software: system software, drivers, real-time embedded systems, etc. Ever wonder why? Mostly, because of the reasons outlined above. Highly vetted code should be easily maintainable. Consistent in handling errors. Deterministic in terms of flow. Suitable for logging and instrumentation. And must strip away abstraction layers that can stand in the way of responsiveness to the real problems. Despite a minority of my colleagues who are, I'm certain, quite diligent about exceptions, most developers aren't consistent, diligent or even paranoid about their code. The majority probably considers exceptions a helpful abstraction intended to "simplify" error handling. The problem is, for these less-disciplined folks, exceptions end up luring developers into the same old trap. The siren-song of, "Don't worry about handling that, you're using exceptions... someone else will take care of it... don't worry...". Exceptions are for... exceptional conditions. Not routine error-handling.

Brad is good enough to list a variety of reasons why he believes exceptions are preferred. We'll recap each one.

"Exceptions Promote API Consistency... the Win32 API is a clear example of this inconsistency through BOOLs, HRESULTS, and GetLastError()". So, given that my operating system, RPC system, device drivers -- on virtually every platform -- generate return codes through either a direct result or a retrieval, how exactly do exceptions provide consistency? No platform supports exceptions at the API level. They are an abstraction tacked on to a return-code model (see #4, above).

"Exceptions are Compatible with Object Oriented Features... yet the developer writing the method has no choice in the return value (if any) of the method". The same problem exists with exceptions. If you plug into an OS feature that's unsupported by your OO language (e.g., a Win32 call you need to make), then you're back in the return-code world. If it's my team writing the code, all of the code we generate will be consistent. Each method will return codes. And they will be checked.

"With Exceptions, Error Handling Code Need not be Near Failing Code..." That's not a good thing. I've illustrated this before and it is illustrated in my example, below. I want error-handling code processing events and unwinding state every step of the way. As if my life depended on it. Because it might.

"...With return-value based error reporting error handling code is always very near to the code that could fail. However, with exception handling the application developer has a choice. Code can be written to catch exceptions near the failure point, or the handling code can be further up in the call stack..." Ahhh, we get to my point #1: Exceptions tacitly encourage abdication of cleanup responsibility. Someone else will NOT take care of it, so don't abdicate the responsibility for the exception!

"With Exceptions, Error Handling Code is More Localized..." Uhhh, yeah. We have to check the return-codes. If you disassemble your exception-handling code, you'll find that, yes, it uses compare op-codes (if statements) to handle the processing of the exceptions. It's just been abstracted out for you, which point #4 addresses.

"Exceptions Are Not Easily Ignored..." Sure they are. Joel's point: "They are invisible in the source code" is highlighted by the frequent servlet stack dumps I receive while using some popular web sites. We've all seen them. Exceptions are easily ignored in a team environment. "Oh, I thought that method was/wasn't going to throw that type of exception". Remember Brad's comment, "...with Exceptions, Error Handling Code Need not be Near Failing Code...". It's called buck-passing. And it's as easy to ignore a critical exception as it is to fail to check a return-code. It's the unwinding logic that's needed, no matter which type of system is used.

"Exceptions Allow for Unhandled Exception Handlers... Today, Microsoft Office uses an unhandled exception handler to gracefully recover and re-launch the application as well as send error information to Microsoft to improve the product." True, if a Microsoft Office product crashes, it can send an error-report and re-launch itself. But my goal is to have zero crashes. A catastrophic exception handler, though, is useful and is not germane to this discussion. This discussion is about handling normal error conditions, not blue-screen-worthy explosions.

"Exceptions Provide Robust Error Info for Logging..." I'm still waiting for someone to tackle the exception version of my tunable logging and instrumentation method, below.

"Handling Errors Consistently and Thoroughly are Essential for Creating a Good User Experience..." True. But this has nothing to do with either exceptions or return-codes. It's about consistently tracking state... and unwinding state when necessary.

"Exceptions Promote Instrumentation..." Again, I'm still waiting for someone to tackle the exception version of my tunable logging and instrumentation method, below.

"Exceptions Unify the Error Model (Hard and Logical Errors)..." Sorry, there are too many types of exceptions... handled differently in too many languages... for this to truly be the case. Maybe someday. And my point #4 stands in stark constrast to this... the fact is, the underlying platforms we work with are all based on return-codes.

And one of my favorite commentators, Raymond Chen pointed out some flaws in Brad's article...

"you don't have to worry about things ending up in partial states"

You still have to worry about partial states - the "catch" and "finally" clauses need to clean up any partial state that may have been caused by an exception thrown while the data structures were unstable.

And how many people wrap their entire function inside a try/finally to clean up partial states?

Exactly. My point #1. For the vast majority, exceptions actively encourage abdication of responsibility for the error.

Now that I've rested my slam-dunk case, I'll declare a tie. It's a tie. It doesn't matter. Diligence about handling errors is the key.

If my life depended on the software (and it probably does, for many of the embedded CPU's in airliners, automobiles, etc.), i would want a meticulous approach to error-handling taken. Every method call checked.

So, consistency -- whether you're using return-codes or exceptions -- is the key. If the only thing that comes out of this debate is that more developers are cognizant of the consistency issue (e.g., the try, catch, finally clauses are always applied... or the do... while structured approach is consistently used), then it will all have been worthwhile.

Return-codes vs. Exceptions, part 417

I can't resist. I am compelled... yes, compelled... to continue the saga of return-codes versus exceptions. Especially since I remembered SOAP. You know, the typical RPC (or even LPC) method of invoking an object. SOAP seems to throw a very large, titanium monkey-wrench into the oh-so-pristine world of exceptions. Consider this RogueWave example of a SOAP call:

In other words, classic RW processing handles the SOAP fault inline, just as it would with a return-code. No exceptions flying around. And that makes sense, given the fact that we have a return-code (or, as SOAP calls it, a fault-code) in the returned XML document.

Let's focus on the Fault element defined in the http://schemas.xmlsoap.org/soap/envelope/namespace. All SOAP servers must always return any error condition in that element, which is always a direct child of the Body element. Without exception, the Fault element must have faultcode and faultstring elements. The faultcode is a code that can identify problems; client-side software uses faultcode for algorithmic processing as the SOAP specification calls it. The SOAP specification defines a small set of fault codes that you can use. The faultstring on the other hand is meant for human consumption.

In other words, when you make calls to objects, you expect to get return-codes back. If you choose to throw exceptions at that point, you can. But it really doesn't serve much of a purpose, since you've got a code (and error text) which tells you everything you need to know to recover gracefully from the fault.

Thursday, June 17, 2004

Clean, simple interfaces

The initial goal when designing a modular system should be simplicity between layers. Making the interface as straightforward as possible will ensure its longevity and encourage its use by others.

Never compromise a design by optimizing for performance up-front. Design the simplest and cleanest interface that has a good chance of being extensible. Odds are extremely high that a clean interface can be enhanced to support very high performance without drastic modification of the design.

Consider the case of a database system. A simplified class might use the calls:

§ Table.Open(table-name, access-mode)

§ Table.RecordAdd(record-data)

§ Table.Close()

If we needed to support the addition of a large number of records to the table (a high-performance bulk load), we might be tempted to add more methods:

§ Table.PrepareForBulkLoad()

§ Table.BulkLoad(record-data)

§ Table.EndBulkLoad()

However, this makes the interface more complicated. What if we preserved the initial implementation and simply required that the caller open the table with a “bulk” access mode when loading large numbers of records?

Because the table has been opened in “bulk mode”, the class knows to cache records before writing them en masse. In other words, all of the optimizations occur inside the class and not in the interface(s) between classes.

The clean, simple design approach permits this to happen without cluttering the design.

Tuesday, June 15, 2004

Exceptions, Tunable Logging and...

I didn't have time to fully explain what I meant by "tunable logging" yesterday. Nor how it integrates with a well thought-out system that handles exceptions gracefully.

Consider an application that you've shipped to a valued customer. The customer calls up one day and says, "Yo, the application isn't working right... all of a sudden the network sharing locks up and no one can open any of the files. We have to have everyone shut down the application and restart it to get it back up!"

What do you do? You could ask for all of their data (could be proprietary, in which case you're out of luck), try to set up a similar network configuration, then simulate their load and activity, etc. Trust me, this is very, very difficult to coordinate and achieve for many classes of application.

Or... if you've employed tunable logging, you can simply have certain key users at the customer site crank up their "log verbosity level" and start generating log-files. Then, when the problem re-occurs, the customer ships you the log-files... which help you diagnose the problem. You quickly turn around a fix, save the day, and the customer remains ecstatic about your product.

So how does tunable logging work? Consider a global "context" object familiar to those who develop servlets, ISAPI extensions, LDAP clients, ... well, just about any significant piece of code. A global context generally consists of settings, methods and such that can be used by the application and its slaves (e.g., multiple threads, forks, and so on). One of the settings in the global context might be:

INT m_nLogVerbosity;

(or its equivalent property). Your application should have a back-door that allows setting this property to between 0 and 9, inclusive. 0 would be the production default, which basically means that only noteworthy errors and events are logged.

If you're writing a web application, your authentication module might set this as a session variable on a per-user basis.

In either case, your application code indirectly uses the verbosity setting. Consider a snippet of code from last time:

1) If the credits table fails to open, this is a critical fault. It gets logged unconditionally (the default argument value is 0, meaning log it no matter the current verbosity level).

2) If the credits table was opened successfully, we will log that fact if the global context's verbosity setting is 5 or higher. Normally, we wouldn't care about this event. But it (and related information like handle info, current user, etc.) might come in handy if we're diagnosing a problem.

3) If the entire transaction succeeded, we'll log it if the verbosity is 3 or higher.

So, in short, we have a logging system that we can "tune" based upon our needs... even after we ship it. Our backdoor allows us to have customers -- in the often-times nasty real world -- turn up the verbosity, capture events of interest, and ship us the logs that summarize critical sequence of events. Result: happy customers. More revenue. You get to buy a Maserati Spyder. What's not to like?

I can hear the whiners now. "What about perrrrrformance? This'll suck the life out of my application! Waaah! Waaaaaah!"

One word: bulls**t. If it's designed correctly, it won't suck the life out of your application. Obviously I don't recommend embedding logging in a highly vetted subsystem like a high-performance sorting routine. But you would put logging in at higher levels (e.g., execution of business rules) that are often delayed due to disk I/O, network activity, database locks, etc.

Consider the overhead implicit in this over-simplified C-style macro:

#define TunableLog(s, v)

if (v >= pCtxt->m_nLogVerbosity) {

Log(s);

}

Hmmm. A single integer comparison. That's the entire run-time overhead. I think we can spare the time, given the advantages with which it provides us.

Back to exceptions for a moment. Consider an integrated, tunable logging system that uses exceptions and not return-codes to handle our snippet. It doesn't appear intuitive to me. But that's just me. I'd be interested in seeing some alternative approaches that use exceptions and tunable logging.

Monday, June 14, 2004

Return-codes vs. Exceptions, part 318

In the never-ending saga/debate/raging conflict, PHP Everywhere's John Lim weighs in with an interesting take:

I totally agree with Doug with regards to real-time software (you want to handle the error as soon as possible with an error code).
However that's not the way I would want to write financial code nowadays. With the relational databases we now use, I'm more likely to throw an exception and perform a database rollback in the exception handler.
In general, technologies that provide timely automated resource cleanup (such as database transactions) make throwing exceptions practical for industrial strength apps.

The question I would ask: how do we draw the line between real-time/mission-critical code and "other" code? How do we gauge whether exceptions or return-codes should be used? Do we simply ask a battery of questions such as:

I contend that if the code is production-ready (in other words, it's not a test harness or utility program), it probably is mission-critical and should provide a tunable logging environment such that faults can be analyzed after the fact. And this goes for user-interface code, not just low-level system services, web services and the like.

Think of a thick-client application: wouldn't it be great to watch the flow of user activity... to find out which screens seem to confuse them, which screens appear intuitive, which screens are heavily used and those that are ignored? Tunable logging, which I consider part and parcel of a proper return-code structure, is a key element of reliable code (at least, in my book).

Note the tunable logging, which I believe to be one of the keys to writing bullet-proof code. The TunableLog method/macro/whatever allows an external verbosity level to control which events are recorded. More on this later.

Sunday, June 13, 2004

Book Review: ENDGAME

Subtitled: The Blueprint for Victory in the War on Terror

Don't read this book because its authors are highly qualified (an Asst. Vice Chief of Staff of the Air Force and a deputy commanding general of the U.S Army, respectively). And don't read it just because their analysis for Fox News has been spot on so far (when others were predicting military disasters in Afghanistan and Iraq, they correctly predicted the military campaigns). Instead, read it as if your life depends on it. Because it probably does.

The authors first describe the gravity of the war against terror. They demonstrate that fundamentalist extremists -- in many countries -- have repeatedly demonstrated that they have no qualms about using any and all means necessary to slaughter innocent civilians. In escalating, nightmarish scenarios, they describe the outcomes of a failure to quickly and completely deal with the "Web of Terror". At the top of the heap, of course, is the very real scenario involving the simultaneous detonation of nuclear weapons in multiple U.S. cities.

The second section of the book describes how we fight: both defensively and offensively. The campaigns in Afghanistan and Iraq are described in excellent detail. And for those who claim that the Iraq campaign was simply a distraction in the war against terror, the authors beg to differ:

"Saddam Hussein's Iraq kept bad company -- as one might expect of a regime that practiced mass executions, torture, and arbitrary arrest and imprisonment and used chemical weapons against its own people. Iraq had extensive dealings with terrorists. Two Palestinians at the top many ... "most wanted" lists in the 1980's and ... 1990's -- Abu Nidal... and Abu Abbas -- were given sanctuary by Saddam Hussein... Nidal led an organization that committed a number of bloody attacks... [and] masterminded the 1985 hijacking of the cruise ship Achille Lauro, a crime that included the murder of [an]... American passenger. Iraq provided training camps for terrorists, most notoriously at Salman Pak... where an obsolescent Boeing 707 was used to train terrorists to hijack airliners. Iraq made cash payments to the families of ... suicide bombers. [Iraq's] vice president... was specifically tasked with supporting... the PLO, Hamas, and ... Islamic Jihad... in addition, the al-Qaeda affiliate, Ansar al-Islam, was based in Northern Iraq..."

Thus, Hussein's massive human rights violations, support of terrorism world-wide, and development and use of WMD's made him a first-tier target in the war on terror. The authors rejected a continuance of the containment policy because sanctions had been subverted by regimes like Syria; international support was wavering; and French, German, and Russion businesses continued to stoke Saddam's massive weapons programs.

The final section of the book discusses how we must wage the war on terror... country by country. In some cases, diplomacy is recommended. In others, quick military action is advocated. The rationale, strategies and tactics are all discussed in compelling detail. Wrapping up, the authors state:

"Our fight, however, is not against a religion. The vast majority of Muslims are obviously peacable people. Our fight is against those who... [would] ... turn Islam into a terrorist creed preaching global violence and revolution. What we can and must do is act against those regimes that train, shelter, and support the terrorists who are bent on killing us (Iran and Syria); put further pressure on Muslim states to curtail radical Islam within their own borders (Egypt, Pakistan); stop Muslim states from subsidizing intolerant religous beliefs around the world (Saudi Arabia); and demand, and bring about, an end to the WMD programs of rogue states (North Korea). We must declare a "no sanctuary for terrorists" policy and enforce it. [We] must be a faithful ally to those Muslim countries that openly oppose Islamist terror and ... stand for tolerance and liberalization... like Morocco, Qatar and Bahrain... [We] must promote economic, political, religious and social freedom through the entire Muslim world..."

It is only in the book's afterword that politics are discussed. The authors state that they have purposely written the book from a military perspective.... not political. However, the Democratic primaries -- in which nearly every candidate denied the existence of a terror network and stated that only "al-Qaeda" is the enemy. The fallacy of that denial is described in great detail throughout the book. The corruption of the UN's "oil-for-food program", the intrinsic monetary links between Hussein and Europe, make relying upon either the UN or conflicted countries like France, Germany and Russia, a foolhardy and dangerous exercise. Unfortunately, the apparent Democratic candidate for President was one of these candidates.

"The Web of Terror will want to influence America's presidential elections because [it] will not survive four more years of George W. Bush." At stake is nothing less than the survival of the United States. Overstatement? The authors clearly demonstrate otherwise. Read this book. And get your friends and family to read it as well. It's simply that important.

Saturday, June 12, 2004

Easter Eggs, Part Deux

It's interesting that you wrote about easter eggs. As an end user and gamer I love them. As an IT professional, I hate them. Since it's clear that you understand the end user "cool" aspects, I thought I'd talk about why I hate them. True story: Back in '98 I was in CITY DELETED for the COMPANY NAME DELETED quarterly meeting. At each of these meetings we have the CIOs of each of the various operating units in for a meeting to discuss new technology trends, standards development, and security. At the time I was the security architect for COMPANY NAME DELETED.

As part of each meeting we invite a strategic vendor to come and talk about their roadmap/vision. At this particular meeting Microsoft presented on the upcoming Windows 2000 release. They gave a very impressive presentation on how MS is now linked to open standards like Kerberos, LDAP, etc. They talked about how secure their new system is going to be. They talked about how they have made major improvements in their QA processes and how we should trust the next version of their OS.

I was interested in easter eggs at the tiem because I had recently discovered the Excel Flight Simulator egg and was really wondering whether having stuff like that in their products effected the stability and security of the product itself. I asked the MS guys from Redmond about the eggs and wanted to know if they underwent the same rigorous QA process as the rest of the product. Since they had just gotten done talking about their new process they had to state that, of course, the eggs underwent the same QA process and we can be assured that they wouldn't impact he security of the product.

Mind you, it took me a good 20 minutes to get them to admit that the eggs existed and were sanctioned by management. The CTO at the time, C, a spicy, southern gentleman, was beside himself. Keep in mind that this guy is all business. COPMANY NAME DELETED as a whole, was having huge problems with performance and capacity on their corporate systems (largely MS-based). Then the vendor comes in and tells you that with every product of theirs that you install, they have included seceral hundred k of unnecessary code. If you multiply this times the thousands of users that COMPANY NAME DELETED had, it adds up to real costs in disk, tape, memory, etc.

It was enjoyable watching the MS reps dance and try to justify why they allow, and in fact endorse, the inclusion of easter eggs in their products. Not only do they exist, but MS knows about every one of them and QA checks them. This final point is something that I have a hard time believing. My point at the time was that they might do better to spend some of the easter egg QA time on making their products more secure and bug-free instead of checking the eggs.

In the end their justification for allowing it was to stay competitive in the market for developers. You see, other companies allow developers to include easter eggs in their products and if MS didn't allow it they would be at a disadvantage when trying to recruit good coders. All in all I have to agree with C. It's bad business. Products are there to meet a specific market need, not stroke the ego of the development teams.

Yo, our egg occupied a total of 17K and was QA'ed by me. Nuff said.

Actually, I'm not saying I disagree with you. Especially if we're talking about Microsoft's products. But since Alpha's were rock-solid from both a security and a stability standpoint ;-) ... there's no reason the development team couldn't engage in a little value-add fun. Consider it the Alpha Bonus Pack.

In praise of continue

I've read some fairly daunting software engineering manuals that discourage -- or even ban altogether -- the use of a 'continue' statement. This, in my experienced but fallible opinion, is a complete mistake. I will illustrate to you, using actual code (no, seriously?), why I believe this is so.

I've abbreviated this code to make it easier to follow. It resides in the heart of the BadBlue scheduler, which mimics Outlook's recurring appointments feature. One of the neat things about the scheduler is that it allows recurring events, no matter how many times it occurs, to be stored in a single record. So if you have an 8:00am daily appointment with your therapist, Monday through Friday, the cost is a single database record.

This code is abridged from the method that finds tasks in a specific "window" of time (a begin-date and end-date). Virtual, recurring events are extrapolated from the physical records that serve as their instantiator. This code finds events (real and virtual) in an ordered list and, if they fall into the window, saves the event ID for the caller's use.

// Set up a search loop until we find something in...
// the find-window (or we run out of items).
//
while (m_posFind != NULL) {

1) Is the loop finished? If so, let the loop terminate gracefully (m_posFind == NULL)
2) Does a "delete" record override this item (e.g., has the appointment been cancelled or otherwise overriden)? If so, skip to the next item
3) Can we find more information about this event in the result set? If not, then it's been removed from the pool of valid events and we can skip to the next item
4) Lastly, is this item even in the window of interest (i.e., does it occur after the begin-date and before the end-date)?

Now imagine if this code was written without benefit of continue statements. In other words, we would use nested "if ... else ..." clauses... and nested rather deeply based upon my initial take.

Now note the relative clarity of the above logic compared to the hypothetical three or four levels of nested if constructs. To me, there's no comparison between the two approaches. I would encourage any developer to use continue constructs wherever clarity can be used to fight nested if statements.

Tuesday, June 08, 2004

Vacation

I will be on hiatus for a few days. Vacation. Beach. Sand. Sun. Kids fighting and complaining.

Actually have read several books, one of which I can highly recommend. Endgame - The Blueprint for Victory in the War on Terror is chilling, fascinating and a very quick read. It spells out exactly what steps the United States (not a multi-lateral commission, the UN or other tools of the proponents of appeasement) must take to survive and win the war on terror.

Friday, June 04, 2004

Easter Egg!

For those unfamiliar with the term "Easter Egg" -- as it applies to software -- it means a hidden feature that can only be activated through an intentional set of actions. Usually, Easter Eggs provide scrolling "credits" where the developers get to pay homage to their friends, hobbies or themselves.

One weekend I decided to create an "ultimate" Easter Egg for Alpha Five version 1. We were getting close to release, so I decided to use the "About Box" as the platform. I wanted to credit the developers and others who played key roles in creating the product. So what would you expect me to do? I created an Alpha Software version of Space Invaders with the developers and other key personnel as the "invaders". That way, a user who got ticked off at some (mis)behavior of the software could exorcise his angst by blasting our heads into clouds of bloody gunk.

How did the Easter Egg get activated? For the answer, I've included an Easter Egg on this page. Roll your mouse over the picture and you'll get the instructions. Not that too many folks still have version 1 installed. It might be in version 2 as well, I can't remember. Senility tends to do that.

Funny story: Richard, Selwyn and I went up to Corel for some discussions about licensing the Alpha Five database to them. While Richard and Selwyn went off into hard-core negotiations with their top people, I showed them the product and some source-code. After a few hours, we'd exhausted those topics. The discussion turned to Easter Eggs. I said, "Hey, what's the Easter Egg in Corel-Draw?" They proudly fired up a copy and unveiled their egg. It depicted a balloon, gently floating, with the names of their key personnel floating by. They turned to me.

I fired up our Easter Egg and played a few rounds, scoring a not-so-impressive 2700 points. They were wide-eyed. "You win! Your Egg rules! We don't deserve to be even considered developers, not compared to you guys." I've kind of forgotten the conversation, though, perhaps that's not verbatim. Anyhow, they did like it and sometimes I think it helped seal the technical side of the deal.

At the time, though, this may have been the premier Easter Egg of consumer software. The Excel team later came out with a mini-"Doom" game with 3D graphics. But this came first and was, as far as I know, the initial foray into a full, playable game in a commercial package.

Thursday, June 03, 2004

Tales of Alpha

There were a lot of benefits to working for a mass-market software company. Some of them were obvious. And some weren't.

The obvious ones probably apply to a fair number of companies. If you wanted to wear a T-shirt and shorts, you did it. If you needed to show up at 11am and work until 9pm, you did it. If you wanted to write a quick program to help with ordering the pizza, you did it.

But some of the benefits weren't so obvious. They were... well, simply, good karma. One day I received this email, which was sent to the entire company.

To: *ALL
From: Peter E
Subject: Re: ted
Date: 10/02/92
Time: 2:39p

by popular demand more on..... Ted "$72000.00 dollar man "
We made his wifes day. He made an pplication for thier girl scout troop.
His application is a run time for the state of tennessee. I believe the application was for the state school
system. A year and a half ago he was a mechanic. Now he is an Alpha applications programmer.

We also made his kids year. This year his kids will have christmas. Next year his family will have a house.
Hats off to all for helping this man with his new life.

peter "proud to be on the team" e...

Ted had created an application for the state and had negotiated a $72,000 license fee using the Alpha run-time engine, which allowed distribution of applications. That was probably one of the better holiday seasons at Alpha.

Bad karma days? I remember one vividly. We had a product for a while called "AlphaWorks" [1], which Cian and another developer had spearheaded. It was a very good product, back in the days of DOS. And it even got a PC Mag Editor's Choice Award in the late 80's or early 90's, tying Microsoft Works.

The company had cut a deal with some low-budget computer firm to bundle AlphaWorks with each computer. The computer was sold on a home shopping TV network with a very, very hard-sell. My guess is the computer company was a fly-by-night operation, because if you received a computer from them, the only phone number you could find in the entire boxed system was... yes, Alpha Software's 800-number on the AlphaWorks manual.

How do I know this? Funny you should ask, friend. I happened to show up at the office on December 26th. I was new to the company and didn't know that day was a holiday. So there was no one there. Ghost town. The phone rang - on night ring, so it echoed throughout the building. It was quite loud and annoying, so I picked it up. This was maybe 9am.

"Alpha Software". Now you need to voice-over a hillbilly [2] accent here.

"Ah jest bought mah comm-pewter and Ah turned it on an' it jest sez Cee". This was in the days of DOS prompts. When you powered on a computer, it did just say "C>".

"Well, sir, we don't make the computer, you'll have to call the "D" company." After several minutes of disentangling myself from this hilljack's situation *, I managed to hang up. NIGHT RING. NIGHT RING.

"Alpha Software, can I help you?".

"Ah jest get me a comm-pewter, when the TV part comes on, it jest sez Cee. What do ah do now?"

Imagine an entire day filled with those calls and you have a good idea how I spent much of 12/26 that year. After a while, fifteen or so calls just like those, I made my home-brew, prison-style earplugs (use warm water, toilet paper and a patient rolling motion) and got back to work.

I don't know if the D company was a mob "bust out" operation or what. But I do know their tech support was nowhere to be found that day. And that's probably how they planned it.

[1] AlphaWorks ended up getting sold to Lotus and it became... drum roll... can you guess... no, the suspense is killing me... hold it... hold it... LotusWorks. Genius! That's gold, Gerry, pure gold! Not sure what happened to it, but the librarian at the Lotus Division of IBM may be able to tell us. IBM must have also purchased the AlphaWorks name, because I believe that's used by one of their mainstream developer sites.

[2] I'm licensed to use and mock hillbilly accents as I've technically been accused of being one. Peace out.

Wednesday, June 02, 2004

$970,000

It's amazing how money is relative. When I was growing up, my Dad served as Treasurer for a volunteer organization. When they held an art auction, he was responsible for all of the funds. I remember him coming back late that night with more cash than I'd ever seen in my life. My brother and I helped him count it. I think it was twenty-two or twenty-three thousand in cash. What surprised me at the time was it really wasn't that bulky. An enormous amount of cash -- in small bills -- could be held in a shoebox.

My Dad had a reputation for honesty that went beyond normal. I assume that's why he had the job. When I was growing up, he was an up-and-coming manager with a large apparel firm. He used to take me into work with him on Saturdays and I'd help him out, pulling swatches to send to the New York office, stuff like that.

Anyhow, I would also work on my baseball magazines and comic books. I used to create my own version of "Baseball Digest" (concentrating heavily on the Reds, since they were the hometown team despite being 150 miles away). And I would also write, illustrate and letter my own comic books: primarily a rip-off of Marvel's Sergeant Fury and Howling Commandos and DC's Sergeant Rock. It was named, uhm, "Sergeant Rage".

Anyhow, I would come in on Saturday with my material ready to be published. And I wanted to use the company's Xerox machine. My Dad had no problem with it. But, scrupulous as he was, he wrote down the number of copies I made so that he could report it and reimburse the company. He even wrote it in the copier log to ensure he didn't forget. Now, that's an honest man.

Years later, he became President of the company and I don't think he needed to worry about logging each copy he made. But he probably did anyway. That's just the way he is.

Back to the relative nature of money: when FASTech closed on its first round of venture money, Jim and Hans sat with the board and the investment bankers, signed all of the thousands of pages of agreements (I was very much a minority shareholder compared to them), and were handed in return a check for $970,000.

Jim said that he was very excited... after all, who wouldn't be with a check for a million bucks in their hands? He ran to the bank to deposit it. Waited in line for a teller. Waited in line a little longer. Finally, walked up to the teller with a grin, expecting a nice reaction. Hell, it's a million-dollar check, we should get at least a smile and a veiled glance from the teller. He made the deposit, watching for a reaction. *Stamp*... clicking on the deposit keys... *Stamp*... not a word, not a bit of reaction from the teller. He gets handed a deposit slip. Have a nice day.

Lesson learned: in Boston's financial district, apparently, a million dollar check ain't diddly squat. It just goes to show you... it's all relative.

Tuesday, June 01, 2004

This will be interesting

In preparation for heading off on vacation this summer, I've configured my blog to accept email posts. Blogger allows you to "check on" an option setting for this purpose. You can then send posts to a secret destination email address hosted by blogger and, if you so choose, they'll be posted and published immediately.

So this is the first test of that system.

As an aside, Blogger has added a feature that allows you to post photos to your blog using your IM client. What's cool is that a "blogger bot" hangs around in your buddy list, waiting for you to message it. When you want to post a photo, you just IM the bot and send it the JPG. Blogger will even host the photo for you (which is a policy change for them, to be sure).

What I haven't figured out is whether you can do this without an IM client. Now that would be very cool.