Archives For October 2010

Following on from the blog post two weeks ago, we’ve made some more improvements. Today, the employees tab in the policy editor gets an overhaul, and is now called the people tab. Here’s a look at what has been upgraded:

New! Multiple final approvers. Whereas we used to limit you to select only one final approver, now you can just construct your report approval/forwarding chains and set the final approvers in the chains to forward to “No one” – this lets us know they are at the end of the chain; the final approver for this chain. Your existing policies will automatically be updated to match this new system.

Multiple administrators. Now there’s no limit on adding other administrators to the policy, and they can now manage everything in the policy: people, categories, tags, limits, etc, and even go on to add other administrators.

Multiple administrators means multiple accountants. Whereas previously, accountant selection was done by a single drop down selection menu, it is now done by just setting a person as one of the administrators.

Forwards-to and submits-to separation. Now each person can be set to submit their reports to one person, and be set to forward their reports to another, so when they don their report-approving hat, their approved reports will be forwarded to a different person, for the next level of approval.

Simplified layout. If they’re in the table, they’re in your policy. No more extra checkboxes for submitters or approvers.

Sync now button for QuickBooks & Google Apps. We’ve added a sync now button which syncs your employee list from QuickBooks or Google Apps, whereas before these people were just (somewhat confusingly) listed in the table even if they weren’t in the policy.

Add person window. This allows you to choose the person’s submit-to and forwards-to settings, gives you more control over sending emails, and makes it more clear that you can add multiple people at once.

Moved CC list. The people on your CC list have been moved into the main table, and set as administrators. All administrators are automatically CC’ed on every report that uses the policy.

As always, let us know what you think of these latest changes, by emailing help@expensify.com, or just commenting here.

I just read this article about how someone made a Firefox extension to steal sessions from popular websites. Are you kidding me? Security isn’t an easy thing, I’ll admit. And maybe we take security to the extreme. But seriously, it’s amazing how many other sites don’t even do the basics. When choosing any service that involves sensitive information, especially sensitive financial information, I’d suggest always looking for the following:

Make sure the address starts with https://. (Sometimes this is replaced with an icon of a little padlock.) This means it’s using the “secure” version of HTTP, which is the protocol that powers the web. Make sure it’s there from they very first page you load, and stays there as you browse the site. Sure, it’s a bit more expensive for the company. But it’s the least we can do.**

Look for PCI compliance. Or, if not that (because it’s pretty intense), at least *some* indication they’re using a third-party approved security framework.

Look for strong partnerships, such as banks and financial institutions. These guys take security really seriously, so if they’re on board, it’s another vote in the site’s favor.

Real security often isn’t easy. But most important things aren’t.

-david

** Note: I should highlight that this blog doesn’t use HTTPS, but it’s also not asking you for anything. When you sign in to Expensify proper — at https://expensify.com — every connection is secure.

(Perfect for new college grads or people who are bored with school and want to get started in the real world!)

Hello, my name is David Barrett and I’m the CEO of Expensify. We do “expense reports that don’t suck!” (Google “expensify” to read more.) We’re getting crushed under an ever-growing pile of super awesome work, and I need one bright soul to help us dig our way out. I can guarantee you fun, an amazing opportunity to learn, and the siren’s call of distant riches. But only if you are *all* of the following:

An incredibly hard worker, even when it’s not so fun. There is a ton of work to do, and a lot of it downright sucks. After all — we do the sucky work so our customers won’t need to. I need you to buck up and grind through server logs, user emails, source code, and bug reports, without complaint or supervision, and come back asking for more.

A cool person to be with. Not a crazy party animal, just someone we can trust, rely upon, hang out with, bounce ideas off of, and generally interact with in a positive way, both personally and professionally. In fact, this is one of the most stringent requirements we have: would you be fun to hang out with day and night on some remote, exotic beach? This isn’t a rhetorical question, either: every year we take the company overseas for a month (on your own dime, sorry) and work incredibly hard while having a ton of fun. We’ve done Thailand, Mexico, India, and Turkey. In fact, I’m writing this from a beach in the Philippines (with an approaching super-typhoon, no less). Where do you want to go next?

Super talented, in a general way. We’re going to throw a ton of work at you of every possible sort, and you need that magic skill of being able to figure it out even if you have no idea where to start. On any given day you might bounce between super low-level coding, super high-level technical support, marketing-driven datamining, updating our user documentation, inventing/designing/building some new feature, etc. This is not a code monkey job — you’re going to be a full participant in the process, and you need to bring your own unique blend of skills to the table.

Specifically talented in a programming way. You can instantly visualize solutions to problems big and small. Your code is always clean, well commented, has good nomenclature and indentation. You can switch on a dime between C++, PHP, Bash, Cron, HTML, CSS, JavaScript, jQuery, Dwoo, SQL — not because you know them all, but because you’re the sort of person who can just pick it up and figure it out. If you’re this sort of person, you’ll know what I mean. If not, then this position isn’t for you.

Bonus points if you’re an SQL data jockey. Not required, but if you love diving into oceans of data and surfacing pearls of wisdom, have we got some exciting things for you.

And there are a bunch more, but odds are if you got this far, nothing I can do would stop you from applying. That’s a problem because while I know *you* are awesome, it’s actually really hard and time consuming to find you in the midst of the literally hundreds of other applications I get from everyone else. So this is where I’m going to ask my first favor: can you make it *really easy* and obvious how great you are, so I don’t accidentally overlook you?

There are probably many ways to do that. But the easiest way is to help me out by answering the following questions:

What’s the URL of your website? If you don’t have one, stop now — please save us both the time by not applying.

When did you start programming? Tell me about your first project, what technologies you used, etc.

Why do you do it? Why programming instead of all the other exciting careers out there?

What was your last/current job, what was/is your total compensation package, and why did you / do you want to leave? Can I have the name and phone number of your last manager? It’s cool if you left on bad terms — I got fired from my last job, after all — just tell me the story.

If you were rich, what would you do, and why?

Without doing any research or asking any friends, what language is each of the following code fragments, and what’s wrong with each (if anything)?.centered { text-align: center; vertical-align: center; }

What’s the biggest, coolest project you ever built from top-to-bottom? Not a component; a whole self-sufficient thing.

Why do you want to work at Expensify, specifically? Not something general about startups overall; what is it about us in particular that interests you?

What’s the catch? Everybody has strings attached — you’ve got something you need to finish first, some big vacation commitment coming up, some particular fear you need addressed or requirement you need satisfied. It’s fine. But what is it?

Bonus: We only hire generalists who can do everything, and nobody is ever pigeonholed. But what would you prefer to work on? Datamining? Customer support? Mobile? System administration? We’ve got it all, let me know what you want.

Please send your answers to dbarrett@expensify.com. If you make an honest attempt at answering the questions above, I promise I’ll respond personally — hopefully in a timely fashion, but definitely sometime.

Thanks. I’m genuinely excited to hear from you. I know there’s someone out there who will be a perfect fit for our team. I really hope it’s you, and I appreciate your help in patience while we figure that out together. Thanks!

Like this:

I was taking with a user the other day and I asked “So, we’ve been doing a ton of work, what do you think?” He thought about it and replied:

I have noticed massive improvements in the iPhone app. With the HTTP application, I’m hard-pressed to think of the changes I’ve encountered.

I was struck in two ways. The first way was: ‘Wow, we’ve been working around the clock making this thing better, and he didn’t even notice?” But then the second was “Actually, that’s great news. That means we’re doing things right.”

See, we have two conflicting goals: make the service better, but do so without alarming or upsetting our existing users. We do this by making very small changes, continuously, and rolling them out in an incremental fashion such that — hopefully — nobody notices. It just gets a little faster, easier, and better every day. Unfortunately (or fortunately?) sometimes we need to make substantial changes that people will definitely notice. We’ve got a few of those coming up, so I’m going to try to explain what they are as they come, starting with the changes we just released to the reports page.

Behold, Stars and Simpler States

Expense reports can go through a surprising variety of states, many of which are generally uninteresting to you but we need to track anyway. But whereas before we just threw the full complexity out there for you soak in, in an effort to make things simpler we’ve opted to reduce the number of visible states from over ten to the following five:

Open – This report is being edited

Archived – The creator has archived the report

Submitted – It has been submitted and is awaiting approval

Approved – It has been “final approved”

Reimbursed – It has been or is in the process of being reimbursed

Some of these states have informational sub-states — eg an “Open” report might have been “[rejected]”. It’s still open, but this gives a sense of how it got there. These sub-states appear in square brackets next to the name of the report. Additionally, for those reports that are waiting on you to do something, we’ve added a helpful “ToDo” column that has a friendly star in it:

So ya, it’s nothing earth shattering. Some people might not have even noticed. But it’s the sort of increasingly-visible change we make every day, rolled out in a way that hopefully minimizes the impact on existing users. We’ve got a lot more on the way, so please speak up if we do something you don’t like or if there’s something you’d like us to do more. Thanks!

Like this:

Hi all, just a quick update to let you know there was a synchronization problem last night that has since been repaired. Affected users have been notified personally; if you haven’t heard anything from us, you’re good to go. More details follow below; please email help@expensify.com or dbarrett@expensify.com if you have any questions or concerns. Sorry for the hiccup — there goes our 99.9% uptime! (Now we’re at 99.84%)

Technical Details

The issue started when we started a large manual update on the database to clean up some old code. Yes, the most dangerous place for a patient to be is on the operating room table, and the riskiest changes are those where nothing is supposed to change. Regardless, the large update became an unexpectedly huge update due to the object in question (policies) being on average larger than we anticipated. This made synchronization take longer than expected between our three realtime replicated datacenters, causing a timeout to occur between the parent database and its children. The children gave up on the parent and continued on in splendid form (the older child became the new parent), and the old parent went into a “something is not right” safe mode waiting for an admin to fix it. All this happened over the span of a few minutes, resolving itself entirely automatically, without any downtime.

However, it left us in a non-ideal state: one of the children was acting as a temporary parent, and the real parent was offline in a safe hibernation mode. Toward that end we copied the database from the new parent to the old parent, and began rebuilding its database from scratch. That takes a long time (we’ve got a lot of data) so while waiting we re-executed the original query by breaking it up into a bunch of smaller queries, all was right in the world. Again, no downtime, no problems.

But things started to get complicated when we brought the old parent back into the fold. Doing a realtime handoff between three live servers — changing which is “in charge” on the fly, without dropping any requests — is difficult. Now this isn’t an unusual operation: we do it all the time. That’s how we upgrade our servers without any downtime, just upgrade them one at a time and the synchronization layer takes care of it seamlessly. Our sync layer is one part of the secret sauce that gives us incredibly high performance and uptime AND maintains incredibly tight security. We’ve done this operation literally hundreds of times before, so often that it’s become routine.

In this case, however, a simple disk error threw us off. To be completely honest, we’re still analyzing that to see exactly why the disk failed in that way at that time. But that actually wasn’t our mistake — disks fail all the time, it’s no big deal. (And any organization that treats it as a big deal obviously doesn’t deal with disks much.) Our mistake was in trying to manually correct the corrupted database rather than just re-copying it and rebuilding it. One of the downsides of doing maintenance like this in the middle of the night is we get tired, and our decisions aren’t always perfect. We should have just let it sit overnight (after all, throughout this whole process the other two databases were performing flawlessly) and dealt with it after a good night’s sleep and a big breakfast.

Unfortunately, we didn’t. So we made what seemed like a fix, ran some tests, and concluded it fixed. (One problem with running realtime replicated databases is that they’re never in *exactly* the same state due to network latency and such. So it’s actually really hard to confirm that two databases are the same without taking them offline, and as you might imagine, we’re loathe to take down the site unnecessarily.) We lit up the parent server, it synchronized and took over parenting from the children, and everything was one big happy family again.

But like most happy families (in the movies, at least), this family harbored a dark secret. The parent had a corrupted soul that slowly infected the children in dark and devious ways. More specifically, new accounts were being improperly created, receipts improperly linked, reports improperly submitted, etc.

The second we discovered this, we took the entire site down for maintenance.

Thankfully the problem had only been in the wild for a few hours. And the problem only affected new users who signed up during that period. But we’re starting to sign up users really fast anymore, so as painful as it is to admit, there were about 400 users whose accounts were affected.

As for how we know this, part of synchronizing is maintaining a log of all changes to the database. Not back to the start of time, but back for long enough such that if two servers disconnect for a bit, they can resynchronize upon reconnect by comparing their journals and see where one left off. Accordingly, this journal is a really powerful debugging tool, and in this case it pinpointed exactly which accounts were affected and in which ways.

So, to make a long story short: not a great day for Expensify. (Though we did use this opportunity to upgrade our servers’ hard drives by 8x, so that’s something.) But we’re back on our feet and have learned a few new lessons. I personally apologize for the problem and, while I can’t pledge that there won’t be more, I do promise that we take every single problem very seriously. Please write help@expensify.com if you feel your account has any problems that we missed, or write me directly at dbarrett@expensify.com if you have any questions or concerns. And if you were one of the very few affected users, have a cup of coffee and send me the expense report — it’s the least I can do.