The log file shows that the timer is set back to 18Hz, which made me ask him on IRC what could be the cause. He identified the "ztdummmy" module (see http://www.voip-info.org/wiki/view/Asterisk+timer+ztdummy). It uses some high-resolution timer apparently which consumes a lot of cpu time to emulate. Unloading that module fixes the problem.