Our Most Recent Posts on Programming

Today we’re excited to announce the latest addition to the Basecamp team: Jay Ohms joins us as our lucky 13th programmer. He’ll be working with our mobile team on Basecamp for Android.

Android enthusiasts will know Jay as the one part of the duo behind Press, the popular Android RSS reader. Press arrived at a time when great design was hard to find on the platform. Jay’s focus on quality and eye for detail made Press a favorite and caught our attention, too.

After spending a week working with our Chicago-based Android team on a trial project we knew Jay, who also happens to live in Chicago (did someone bribe an alderman?), was a great fit. Jay makes us better on Android and we’re excited to show you what we’ve got in store for 2015.

Of the many axioms in the world of software, few rise to the occasion of Thou Shall Not Rewrite Your Application. Spolsky called it the “single worst strategic mistake that any software company can make” in his seminal 2000 essay Things You Should Never Do.

The Big Rewrite has been likened to all sorts of terrible things, but perhaps the most damning is its association with declaring technical bankruptcy. Basically, the only time it’s reasonable to embark on the terribleness of the rewrite, is when you’ve been profligate cowboy coder. When your mountain of technical debt is crashing down upon you.

So it’s the white flag, then. It’s giving up. Capitulation! The error of your inadequacy has finally caught up and will be taking you to the cleaners. Who the hell wants to be that sorry sob of a programmer!?

No wonder programmers are loathe to question the wisdom of NEVER REWRITE. Either they’ll reveal their own morally deficient ways, or they’ll be seen as apologists for others traveling that deviant path.

Now, axioms develop for a reason. Many a rewrite has been started and gone astray for all the wrong reasons. Either it truly was a result of technical bankruptcy, or, perhaps even worse, it was a result of perceived technical bankruptcy by a new team uninterested in learning why things became the way they are. As Spolsky quips, such a rewrite is basically the same software with more and different bugs!

But there are other types of rewrites. The one most dear to me is the “Don’t Try To Turn A Chair Into A Table” rewrite. It’s the one we committed when we launched the new version of Basecamp a couple of years ago. A full, start-over, everything-is-reimplemented rewrite of Basecamp because we wanted it to do different things.

Yes, Basecamp Classic and the new Basecamp both juggle many of the same ingredients of project management and collaboration, but they’re mixed together in very different curations. So while we could have gotten from A to B one carefully tested refactoring and commit at the time, it would have been a fool’s errand.

Starting over allowed us to question the fundamentals of how the application worked and how it was implemented. We were able to make leaps, not just skips.

A chair can indeed be turned into a table with enough effort. Both have four legs, and all you need to do is chop off the back of the chair, and somehow refasten it to the base to extend the surface. Oh, and maybe some new wood to raise the legs up higher. It can be done, but why on earth would you?

So we decided to leave the chair alone. People were using the chair, and still are – years after the new table premiered! And we got to build a beautiful new table that’s been serving us so very well for the last couple of years.

That’s really the bottom line here: We rewrote Basecamp from scratch and it turned out great! Better than great, actually. We got to leave well enough alone for people who had adopted and grown accustomed to Basecamp Classic, we got to offer a brand-new product to new customers that was the very best we knew how to make, and we got a greenfield codebase to enjoy as well. Plus-plus, would do again!

So what am I saying here? Rewrite willy nilly whenever you get blue over seeing a few modes of progress made cumbersome because of poor past decisions? Of course not. Embarking on The Big Rewrite is not a choice to be made lightly, but a choice it remains. It should be one of the options on the table.

If you’ve accumulated enough fresh ideas about how your application can be radically different and better, The Big Rewrite may very well be just what the carpenter ordered.

We’re looking for a programmer to lead us on Android and push us further into the platform. This is a unique opportunity: we don’t hire often! We’ve got a serious foothold in the ecosystem, and we want you to take us to the next level.

We’re happy to welcome applicants based anywhere around the world. Our office is based in Chicago, but our team is spread out over multiple countries and time zones. You can work from anywhere! We’re so serious about working remotely that we even wrote a book about it.

Basecamp offers benefits for the long run, including: 4 day work weeks during the summer, an exercise and CSA stipend, a 1 month sabbatical every 3 years, maternity and paternity leave, and health insurance that covers children, marriages, and domestic partnerships.

Who are we looking for?

We’re looking for someone who lives and breathes mobile, and especially on the Android platform. Fluency in Android’s patterns and APIs is a must. (Half-hearted iOS ports drive us crazy too!) We would love to talk to a developer who has experience shipping applications with a team, or independently on your own projects.

Here are some examples of day to day interactions with our mobile team. One day we might be diving through the Material design spec…

...and the next day we’ll be tossing around ideas for an Android Wear prototype.

What will you work on?

You’ll work on our mobile team to support and improve Basecamp for Android, side by side with our other programmers, designers, and QA team that enable our customers to get their work done better on the run.

Here are some fun stats from our app’s usage in the wild. Android is our fastest growing mobile application right now:

Like many others with Android apps, we see a lot of older versions of Android, but we’re not afraid to welcome those users in while experimenting with the latest goodies from Google:

You’ll join the our team of programmers who work on our Rails and iOS apps. Basecamp is the home of Ruby on Rails, and we’ll be happy to introduce you to our infrastructure. Part of being a programmer at Basecamp is our on-call rotation, where we help fix day-to-day issues for our customers using our products.

We also believe strongly in putting everyone on support, which usually happens around once a month. If you have some experience with Rails, that’s a bonus—but not a requirement.

How can you apply?

We love cover letters. They’re a great way to introduce yourself, but don’t be afraid to surprise us in other creative ways too. We want to see what you’ve built and shipped, so any existing applications on the Google Play Store that we could try out would be great to include.

Our setup at Basecamp for testing our iOS apps is pretty neat. It (synonymously) involves Xcode’s CI Bots, but that’s easily the most uninteresting part of it. Here’s how we continuously integrate our iPhone and iPad apps at Basecamp and get build results into our Campfire.

Build your bot

This is easily the trickiest part. However, it is pretty boring. Once you’ve followed Apple’s tutorials to setup OSX Server and the Xcode service (which I will skip or lose pretty much everyone reading this for several hours), you just need to head to Product > Create Bot inside of Xcode.

I recommend creating an empty Xcode project first to just make sure everything is working – don’t start with your main iOS codebase. I named it “Placebo”.

We have some specific settings on our bots. Here’s a screenshot for each screen in the bot wizard, since there’s no way to share them outside of the Xcode walled garden. The first screen is the easiest:

Almost 2 years ago I wrote about our approach to building our first in-house native application. This week, we shipped version 2.0 of Basecamp for iPhone, which now shares a codebase with Basecamp for iPad. I’m proud to say this is our best iOS release yet. Give it a shot!

Here’s the last 2 years worth of iOS and Android usage by operating system version for Basecamp, combining our native app usage and our mobile web views:

Fragmentation is the past for Android, but now it’s the present for both of the major mobile operating systems. My big question: will we see either platform be able to converge in the future, or is this the new normal?

Much of software development planning is done through estimates. You give me a description of the feature, I give you a best guess on how long it’s going to take. This model has been broken since the dawn of computer programming, yet we keep thinking it’s going to work. That’s one definition of insanity.

What I’ve found to be a more useful model is simply to state what something is worth. If a feature is worth 5 weeks of development, for example, that’s the budget. Such a budget might well be informed by an estimate of whether some version of that feature can be possibly built in 5 weeks, but it’s not driven by it.

Because most features have scales of implementation that are world’s apart. One version of the feature might take 2 weeks, another might take 6 months. It’s all in where you draw the line, how comprehensive you want to be, and what you’re going to do about all those inevitable edge cases.

The standard response to the estimation approach is to propose a 100% implementation that’s going to take 100% of the effort to build. Some times that’s what you need. Nothing less than having everything is going to be good enough. I find that’s a rare case.

A more common case is that you can get 80% of the feature for 20% of the effort. Which in turn means that you can get five 80% features, improvements, or fixes for the price of one 100% implementation. When you look at it like that, it’s often clear that you’d rather get more done, even if it isn’t as polished.

This is particularly true if you don’t have all the money and all the people in the world. When you’re trying to make progress on a constrained budget, you have to pinch your development pennies. If you splurge on gold-plating for every feature, there’s not going to be anything left over to actually ship the damn thing.

That’s what proposing a budget based on worth helps you with. It focuses the mind on what assumptions we can challenge or even ignore. If we only have 5 weeks to do something, it’s just not going to work to go through the swamp to get there. We have to find a well-paved road.

In the moment, though, it can be frustrating. If we just had a little more time, we could do so much better! So much better for whom? Your developer pride? Or the customer? Will the latter actually care about all the spit and grit you poured into these particular corners? Don’t be so sure.

In the end, accepting a budget is about accepting constraints. Here are the borders of scope for our wild dreams and crazy colors. Much of invention lies in the fight within those constraints. Embrace that.

When we launched the iPhone version of Basecamp in February of last year, it was after many rounds of experimentation on the architectural direction. The default route suggested by most when we got started back in early/mid-2012 was simple enough: Everything Be Native!

Thus their decision was likely driven by what the state of the art in HTML and on mobile looked like circa 2010-2011. In early 2010, people were rocking either the iPhone 3GS or 3G. By modern 2014 standards, these phones are desperately slow. Hence, any architectural decisions based in the speed of those phones are horribly outdated.

Performance tuning is a fun sport, but how you’re keeping score matters more than you think, if winning is to have real impact. When it comes to web applications, the first mistake is start with what’s the easiest to measure: server-side generation times.

In Rails, that’s the almighty X-Runtime header — reported to the 6th decimal of a second, for that extra punch of authority. A clear target, easily measured, and in that safe realm of your own code to make it appear fully controllable and scientific. But what good is saving off milliseconds for a 50ms internal target, if your shit (or non-existent!) CDNs are costing you seconds in New Zealand? Pounds, not pennies, is where the wealth is.

Yet that’s still the easy, level one, part of the answer: Don’t worry too much about your internal performance metrics until you’ve cared enough about the full stack of SSL termination overhead, CDN optimization, JS/CSS asset minimization, and client-side computational overhead (the latter easily catching out people following the “just do a server-side API”, since the json may well generate in 50ms, but then the client-side computation takes a full second on the below-average device — doh!).

Level two, once reasonable efforts have been made to trim the fat around the X-Runtime itself, is getting some big numbers up on the board: Mean and the 90th percentile. Those really are great places to start. If your mean is an embarrassing 500ms+, well, then you have some serious, fundamental problems that need fixing, which will benefit everyone using your app. Get to it.

Keep going beyond even the 99th

Just don’t stop there. Neither at the mean or the 90th. Don’t even stop at the 99th! At Basecamp, we sorta fell into that trap for a while. Our means were looking pretty at around 60ms, the 90th was 200ms, and even the 99th was a respectable 700ms. Victory, right?

Well, victory for the requests that fell into the 1st to 99th percentile. But when you process about fifty million requests a day, there’s still an awful lot of requests hidden on the far side of the 99th. And there, young ones, is where the dragons lie.

A while back we started shining the light into that cave. And even while I expected there to be dragons, I was still shocked at just how large and plentiful they were at our scale. Just 0.4% of requests took 1-2 seconds to resolve, but that’s still a shockingly 200,000 requests when you’re doing those fifty million requests.

Yet it gets worse. Just 0.0025% of requests took 10-30 seconds, but that’s still a whooping 1,250 requests. While some of those come from API requests that users do not see directly, a fair slice is indeed from real, impatient human beings. That’s just embarrassing! And a far, far away land from that pretty picture painted by the 60ms mean. Ugh.

Finally, there was the true elite: The 0.0001%, for a total of 50 instances. Those guys sat and waited between 30 and 60 seconds on their merry request to complete. Triple ugh.

Dragon slaying

Since lighting the cave, we’ve already been pointed to big, obvious holes in our setup that we weren’t looking at before. One simple example was file uploads: We’d stage files in one area, then copy them over to their final resting place as part of the record creation process. That’s no problem when it’s a couple of 10MB audio files, but try that again with 20 400MB video files — it takes a while! So now we stage straight in the final resting place, and cut out the copy process. Voila: Lots of dragons dead.

There’s still much more work to do. Not just because it sucks for the people who actually hit those monster requests, but also because it’s a real drain on the rest of the system. Maybe it’s a N+1 case that really only appears under very special circumstances, but every time the request hits, it’s still an onslaught on the database, and everyone else’s fast queries might well be slowed down as a result.

But it really does also just suck for those who actually have to sit through a 30 second request. It doesn’t really help them very much to know that everyone else is having a good time. In fact, that might just piss them off.

It’s like going to the lobby of your hotel to complain about the cockroaches, and then seeing the smug smile of the desk clerk saying “oh, don’t worry about that, none of our other 499 guests have that problem… just deal with it”. You wouldn’t come back next Summer.

So do have a look at the far side of your histogram. And use actual request counts, not just feel-good percentiles.

Much intellectual capital is spent examining the logical advantages and disadvantages of our programming tools. Much ego invested in becoming completely objective examiners of productivity. The exalted ideal: To have no emotional connection to the workbench.

Hogs and wash. There is no shame in being inspired by your tools. There is no shame in falling in love with your tools. Nobody would chastise a musician for clinging to their favorite, out-dated, beat-up guitar for that impossible to explain “special” sound. Some authors even still write their manuscripts on actual type writers, just for the love of it.

This highlights the tension between programmers as either engineers or craftsmen. A false dichotomy, but a prevalent one. It’s entirely possible to dip inspiration and practice from both cans.

I understand where it’s coming from, of course—strong emotions often run counter to good arguments. It’s hard to convince people who’ve declared their admiration or love of something otherwise. Foolhardy, even. It can make other types of progress harder. If we all fell madly in love with Fortran and punch cards, would that still be the state of the art?

I find the benefits far outweigh the risks, though. We don’t have to declare our eternal fidelity to our tools for them to serve as our muse in the moment. And in that moment, we can enjoy the jolt of energy that can come from using a tool fitting your hand or mind just right. It’s exhilarating.

So much so that it’s worth accepting the limitations of your understanding. Why do I enjoy Ruby so very much? Well, there’s a laundry list of specific features and values to point to, but that still wouldn’t add up to the total sum. I’ve stopped questioning it constantly, and instead just embraced it.

Realizing that it’s not entirely rational, or explainable, also frees you from necessarily having to push your muse unto others. It’s understandable to be proud and interested in inviting others to share in your wonder, but mainly if they haven’t already found their own.

If someone is already beholden to Python, and you can sense that glow, then trying to talk them into Ruby isn’t going to get you anywhere. Just be happy that they too found their workbench muse.

At the end of the day, nobody should tell you how to feel about your tools (let alone police it out of you, under the guise of what’s proper for an engineer). There’s no medal for appearances, only great work.