Cedar Build Process

Since Heroku runs on
Heroku, after
receiving a git push of an application, the build request is dispatched to
a regular Heroku app named Codon that handles builds. Codon runs
a buildpack which compiles
the application so that it can be deployed.

Normally apps running on Heroku are entirely isolated using Linux
Containers, but to perform builds Codon runs
untrusted code in this container.

Source Code Exposure

I encountered a Ruby exception and backtrace from the Heroku build system while
experimenting with custom
buildpacks. Ruby
backtraces look like this:

Backtraces include the paths to the source files that encountered the exception.
This pointed me to the source files for Codon, which indicated the possibility
of gaining read access to the code.

I then ran a custom buildpack that copied the source code into my Heroku app and
verified that it was possible to view the source code of Codon.

While examining the source code I discovered that there was another
vulnerability that was much more serious than source code exposure.

Sensitive Credential Exposure

Like most Heroku apps, Codon uses environment
variables to configure
runtime options including sensitive credentials. This ensures that credentials
are not checked into version control. However, due to the constraints of Heroku
containers, Codon is running as the same user as the buildpack, which is
untrusted. This allows the buildpack to dump the environment variables of Codon
from the Linux process table:

cat /proc/*/environ

The environment variables exposed included critical credentials such as internal
API keys, a SSH private key with access to source code repositories, Redis
connection details, and a key with access to their
Campfire account.

Disclosure Timeline

Immediately after discovering this vulnerability, I sent an email to Heroku’s
security team to start the disclosure process. I requested a PGP key first, as
they did not provide one on their website. Here is the discovery and disclosure
timeline:

2012-06-26 19:45 PDT

Encountered backtrace and began experimenting.

2012-06-26 20:25 PDT

Sent email to Heroku asking for PGP key.

2012-06-26 22:40 PDT

Received PGP key from Heroku.

2012-06-26 22:56 PDT

Received follow-up email with mobile phone number of a Heroku security engineer.

2012-06-26 22:58 PDT

Sent PGP encrypted description of the vulnerability.

2012-06-26 23:06 PDT

Received confirmation of receipt.

2012-06-27 12:01 PDT

Received confirmation that an interim patch would be pushed in a few hours, and full patch by Tuesday (2012-07-03).

2012-06-28 20:44 PDT

Checked validity of credentials, SSH and Campfire keys were still valid.

2012-06-29 16:13 PDT

Checked validity of credentials, all credentials were invalid.

2012-07-03 13:35 PDT

Received confirmation that the issue had been patched.

Customer Impact

The build system appears to have been vulnerable since the Cedar stack
launched about
a year ago. Customer applications and credentials could have been compromised at
some point due to the credential exposed by the vulnerability. Anyone who ran
applications on Heroku during this period should immediately reset all sensitive
credentials, and audit their access logs to determine if any infrastructure or
data has been accessed.

I suspect that a variant of this vulnerability may exist in other Platform as
a Service build systems. Further research is warranted.

Full Disclosure: I remain a Heroku customer with several apps in
production, and I have no plans to change platforms. Heroku offered me a paid
penetration test contract, but required that I sign a retroactive non-disclosure
agreement which would have precluded publishing this article.