That would explain the jump in load on the api (bug 1150631) - guess I should perform some further analysis and ban IPs if needed. We should also probably set an API limit/threshold for reads too (we have a limit already, but presuming it's for submissions only - or else the threshold is too low).

camd, jgfiffin, just saw this:
19:03 camd jgriffin: so, I think based on this, there's nothing immediate that needs to happen: https://rpm.newrelic.com/accounts/677903/incidents
19:03 camd all the recent alerts were not error rate changes, like on 3/31.
19:03 jgriffin I agree
19:03 camd just disk and cpu space issues that are hovering.
I may be misunderstanding, but neither the disk space alerts (this bug) nor the cpu issues in bug 1150631 are us just hovering near a threshold we've been at for a while, there was a distinct change in traffic (likely from something abusing or api) and the only reason the disk usage alert closed was because fubar kindly switched on log compression. See comment 2 onwards :-)