Heartbleed and Harvest

On Monday, April 7th, there was a update released for the OpenSSL library to address security vulnerability CVE-2014-0160, more commonly known as the Heartbleed bug.

OpenSSL is widely used by many websites, including Harvest, to securely and privately transmit data on the Internet. The information exposed by the Heartbleed bug could allow an attacker to eavesdrop on these communications and steal data that could be tampered with or used to impersonate users.

Since the announcement, we have upgraded all of our infrastructure and Harvest is no longer vulnerable to Heartbleed.

We have no evidence that this exploit was used against Harvest. However, the nature of this attack makes detection very difficult, so we are being very cautious and aggressively updating anything that may have been compromised.

What you should do

In order to protect your account, you should do the following as soon as possible:

Revoke your access tokens for authorized applications. On the same Security page as step 1, you may see an “Authorized Applications” section. You should revoke any tokens listed there, as they have the same access to your account as your password. Revoking these tokens will log you out of any applications listed (such as Harvest for iPhone or Harvest for Mac).

What we’ve done

We upgraded OpenSSL for all of our main web application servers within minutes of the official announcement on April 7th.

One server, our internal infrastructure management tool which does not contain customer data, was not upgraded for three days because we had to wait for a new software release from an external vendor. This last server was upgraded on April 10th.

Once all of our systems were upgraded, we were no longer vulnerable to an attacker collecting any new data, but pre-existing things like the private keys for our SSL certificates could have been previously stolen. To mitigate this threat, we regenerated new private keys for all of our servers and had all of our SSL certificates reissued.

Unfortunately, this already-lengthy process took longer than anticipated, because we had to rely on a vendor to reissue our new SSL certificates, and this vendor introduced a bug into the certificate-issuing process that resulted in many instances of faulty certificates being issued. This bug impacted a number of their customers in addition to Harvest. We ultimately had to select a new vendor.

Lastly, we reset all user sessions to expire any sessions that may have been hijacked during the vulnerability window. You may have been logged out of Harvest and forced to log back in — that was a side effect of this reset.

While upgrading our infrastructure due to Heartbleed, we also hit an unrelated operating system bug which caused two brief Harvest outages. This made the introduction of the upgraded software and the new SSL certiticates impactful to customers. We are truly sorry for this interruption.

That was an excellent and honest post about the actions you’ve taken and the unforeseen impediments you encountered, T.J. It contracts starkly with some other (much larger) companies we critically rely on.

If only all businesses affected by HeartBleed were as clear and transparent! Thanks guys.