There's a pattern to disasters: When you examine the post mortems, its not just one failure, but a cascade of multiple failures that overwhelm the system's safety features. Pushing to a red build risks piling one failure upon another.

Today I learned that there’s a difference between a throw-away class and a sacrificial class. When writing rspec tests, you want the latter; otherwise you might create some very hard-to-debug test pollution.

⊕https://www.flickr.com/photos/tpapi/2765541278Memory leaks happen. They shouldn’t, but you just can’t squash enough of them. Or the leak is so slow that it takes hours to manifest. The trouble with memory leaks is that they are permanent for the life of a process. Once a process has expanded its VSZ, there’s no way in Unix to shrink it. So your process starts to bloat, eventually your servers page and thrash. Performance tanks. Then your pager starts going off and there goes your beauty sleep.

I’ve written my fair share of apologies. It’s a skill, like any other, one that you wish you didn’t have to practice. Yesterday, I had to write a mea culpa to our users. It was our fault, and we came clean about it. I got compliments on the letter! I think I increased customer loyalty as a result. Fuck-ups don’t have to hurt your business.

I am often asked about the meaning of things in Pivotal Tracker, such as “What’s the difference between a bug and a chore?” Or, “How can I tell the difference between a bug and a feature?” Another oldie but a goodie is, “What story should I pick from the backlog?” Here are my biased answers.

For any sufficiently large application, you want to minimize interruptions to service while deploying new code. This is especially challenging for migrations on Heroku and other Platforms-as-a-Service, where you have a Catch-22 problem; you want to run your migrations, first, but you can’t run them until you’ve deployed the migrations to production. Usually, at this point, your new migration-dependent features are in the deploy, too. So when your application restarts, and the migration hasn’t run, yet, your poor application will trip some exceptions, and perhaps, create problems for your users. Of course, you could put the app into maintenance mode, but that creates more downtime for your users.

Once a week, we run a query on our production system that takes a very long time to complete. Invariably, it times out because the default settings on Heroku are (quite reasonably) for web / API interactions; anything longer than 30 seconds is too long and the connection should be closed to let other requests through. How do you get around this, then? You could set up your database.yml file so that you connect to your production database from a developer workstation. This is actually quite useful, but opens up a hellacious security hole. Let’s not get started about the consequences of accidentally running rake db:drop:all! Moreover, you want your system to be automated as much as possible, and that means running your tasks on a Heroku instance. Fortunately, with a little hackery, you can modify the connection timeout settings within your application. Do this carefully, and make sure that it can not leak into your running web application dynos.

On my current project, we're using RabbitMQ. It's a bit of infrastructure that has to be present, and if it isn't, our integration tests will fail with mysterious error messages. We want our tests to be informative, so let's write a test that asserts that we have the requisite infrastructure in place.

If you've written enough integration tests (with Capybara et al.), you must have noticed how much time your tests spend just logging into your web app. Even if it takes 1 second each time, it starts to add up. Here's a solution that I've written several times, now. I create a test "jig" that allows me to authenticate into my application with a single visit.

A few weeks ago, some of the managers got together for drinks after work. I started to reminisce about an old employer who had, what I thought, was an incredibly unique and reasonable way to handle the “vacation liability” problem. The old firm was a consulting firm, mostly military contracts, in the DC area. We were paid for every hour we billed, and we accrued vacation hours for every hour we were paid. It created an environment where the top performers could rack up so much vacation that they could not use it all. Many companies handle this with the familiar “use it or lose it” policy, which caps the accrued vacation hours to some limit. Instead, the firm let vacation accrue forever, with one modification: When you got a raise, your vacation hours were reduced by exactly the same percentage.

It is a time-honored tradition for Pivots to blog about their first few months at Pivotal. A typical day at Pivotal is strong work. It’s different from any previous job. It’s exhausting. After six weeks or so, however, the Pivots find their rhythm. I’m not going to write any more about that. I’ll include some links, below, so you can read them for yourself. I’m here to write about the two-years-later; When most developers are itching to move on to the next big thing. I’m still happily learning new stuff every day.

Part 4 Introduced PhantomJS as an easy and faster alternative to headful Jasmine testing. Part 3 added jasmine-ajax so we can verify that stores and models react properly to back-end data. We also learned how to use stores to test views, without depending on a back-end server. In Part 2 I showed you how to unit test Sencha model classes in Jasmine. In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework.

Part 3 added jasmine-ajax so we can verify that stores and models react properly to back-end data. We also learned how to use stores to test views, without depending on a back-end server. In Part 2 I showed you how to unit test Sencha model classes in Jasmine. In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework.

In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework. In Part 2 I showed you to unit test Sencha model classes in Jasmine. In this part, we’ll test “views” and show how to mock data stores.

In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework. We’re going to take a bit of a breather from all the hard work we did last week. In this blog, I’m going to show you how to test simple models.

We built an(other) object factory module for our current project and it looks a lot like all the others: After a while, we noticed that the create methods were all exactly the same. Time for some dynamic refactoring!

Volatility is what Pivotal Tracker uses to measure the consistency of your team’s work output. You can use that number to help you estimate the first approximation to answer the eternal question, “Will I make the deadline?”

One of the many pleasures of working at Pivotal Labs is that we are encouraged to release some of our work as open source. Often during the course of our engagements, we write code that might have wide-spread use. Due to the nature of our contracts, we can not unilaterally release such code. Those rights belong to the client. And rightly so. So, it is an even greater pleasure when one of our clients believes in "giving back" to the community, as well.

On my current project, we needed to prove that an action cache was working as expected. Alas, the blogosphere had either out-of-date or unhelpful information. So, after many experiments, we came up with an RSpec test that does what we want. It seems ugly to me, and I hope there's a better way. The names have been changed to protect the guilty. Any resemblances to actual classes and methods are purely coincidental.

One of the thornier problems in our workflow is knowing when assets are delivered from the designer and keeping them in sync with our application as they change. We used to use e-mail, Skype or sticky notes. The trouble is that the designer's file naming and directory structure were never quite the same as the application's /public/images directory, so direct comparisons were impossible and we ended with a lot bookkeeping to make sure that we didn't lose any changes. Our solution is to clone the project's git repository into a folder inside a shared Dropbox folder.

The purpose of the "speech" is to condense the project definition
into something that would fit into the time it takes to ride an
elevator. This is where all good ideas are pitched, apparently; you
stalk an unwary VC until he or she is trapped inside the elevator has
no choice but to listen to your prattling. If your idea has any merit,
pray that the VC doesn't call on someone else to launch the project
and leave you out in the cold.

Let's assume that you've adopted the techniques described in Using CVS for Web Development
(or [Dreilinger]) and you
are
the Release Master. Now imagine that instead of one web server you have a
web server farm of 10 or 100. How do you prevent chaos and carpal tunnel
syndrome from keeping you chained at your desk for days on end? Does this
look familiar:

Moore's
Law has had some unintended side effects: What to do with all that
raw power sitting next to, on top of, or behind your desktop
(or laptop or Mazda
Miata). Most people fritter it away on silly things like
screen savers, those flying toasters from hell. There are, however, a few, but
growing number that are putting idle hardware into "productive" use.

I've been thinking about the next logical step in Silicon Valley human
resource management. Already there are campuses where you never have to
leave for (almost) all of your personal needs: the dry-cleaner will pick
up and deliver; cafeterias, webvan.com and take-out-taxi/ meals on wheels
keep you fed; health clubs, masseuse and the roving chiropractor keep your
body functional; showers and locker rooms keep personal hygiene up to
acceptable standards. So what's next in this reinvention of the company
town?

It’s hard to believe that it’s been over a year since we left the
house in Los Gatos and moved aboard the S/V Wishful Thinking.
We made trips to Angel Island, South Beach Harbor, Richmond Marina,
Pier 39, Vallejo YC, Golden Gate YC… covering something over 800
miles this year.

Day 1: Moved in to my new digitally-maxed out Hunter 450 at last. Finally, I
live on the Smartest Boat in the Marina. Everything's networked. The
cable TV is connected to our phone, which is connected to my personal
computer, which is connected to the shore-power lines, all the appliances
and the security system. Everything runs off a universal remote with the
friendliest interface I've ever used. Programming is a snap.

An eventful weekend, that's what we had. It started Friday evening
about 6 PM as that was our window of opportunity to escape from the
shallows of the marina entrance in the South bay. The winds had been
blowing 25+ out of the north since early afternoon so it was going to
be wet and splashy, but we really wanted to stay at Treasure Island
Marina that night and on to Angel Island early the next morning. So,
it was now or never. Wishful Thinking was actually
the last of three to leave. Gypsy, a Union 38, left
about 5 PM and Dragon Lady, a Cheoy Lee 40, left
about 5:30. We gave them a call once we were underway. "Hope you
guys are reefed" came their answer. "Reefed" I said, "hell we're
motoring!" "We're making fine progress just quartering these seas a
bit and heading into the 25+ knots".