The instance has some local disk space allocated under /srv/ (via puppet class role::labs::lvm::srv ). Would be a nice destination for core files which would be local to the instance and avoid filling the NFS shared disk space.

The latest production puppet code for setting up hhvm moves the cores to /var/log/hhvm. We need to get deployment-mediawiki02 running puppet again and then probably make /var/log/hhvm a symlink to /data/project/logs/hhvm to ensure that we have lots of space for cores. This is of course only useful if we have someone watching for hhvm crashes and doing something to triage the bugs that cause them.

Note I originally created the bug because hhvm send cores to /tmp/ which should be configured in the hhvm conf file to point somewhere else, or become configurable (I think the path is hardcoded in hhvm).

Note I originally created the bug because hhvm send cores to /tmp/ which
should be configured in the hhvm conf file to point somewhere else, or
become configurable (I think the path is hardcoded in hhvm).

FYI, by default, the linux kernel creates core files in the process's CWD. If you want to retain the core files just not in /tmp, you can give a file pattern (including path) in /proc/sys/kernel/core_pattern

FYI, by default, the linux kernel creates core files in the process's CWD.
If you want to retain the core files just not in /tmp, you can give a file
pattern (including path) in /proc/sys/kernel/core_pattern

Right. That can easily be done in the pre-start stanza of the upstart job. But I don't agree with Antoine and Bryan that we should, in fact, do this. If it's important to retain core files, then let's keep them in /tmp. If the beta cluster app servers don't have enough space in /tmp, then that's the actual bug, and we should fix it by making sure they do.

Gratuitous and unprincipled divergence from production compromises both the beta cluster and production: the beta cluster because its fidelity to production is its very value and purpose, and production because the Puppet change needed to make the divergence possible means adding a useless knob to the manifests. Sometimes it's unavoidable, but I don't think this is one of those times.