Static Publishing - memory issue - FileSystemPublisher/Director

I've got some serious performance problems with Silverstripe so I hoped to find a solution in static publishing my site. When performing .../dev/buildcache?flush=1 I get an 'out of memory' error.

Because I don't understand why such a simple action should take lots of memory I tried to analyse the code. It seems that the following line in cms/code/statispublisher/FileSystemPublisher.php is responsible for the increase in memory:
$response = Director::test($url);
This lines eventualy calls Director::handleRequest(...) so the real issue goes deeper.

You can see what's happening if you just call this method multiple times for the same url. Then you see the memory usage grow:
for($i=0;$i<100;$i++) {
echo "--- iteration $i with memory usage: " , memory_get_usage() , "\n";
$response = Director::test($url);
}
break;