ok, in the controller action, we build an array of rows that have 100+ columns (legacy ftw) and join quite a few tables. Not too mention there's a couple of thousand rows. That's using up a large chunk of memory, and is partly responsible for the massive memory use.

In the view, we loop through the array, convert it to csv, and append it to the $output scalar. Then, once we're finished, we write the whole thing out via $c->res->body. This is another very memory intensive operation, and since the massive array is still in scope, it all adds up to about 1.1gb, that is never freed back the os, because that's what perl does. And I don't know about you, but when i see 1.1gb being used for no reason, I think that's bug in my code.

Solution

There's two problems, so lets fix them one at a time

The view

Creating each row and appending it to a string is very memory expensive, but luckily for us Catalyst::Response has a method that will allow us to stream the output, row by row. so we change the lines

$output .= $csv->string;

to

$c->res->write( $csv->string );

we have already saved a massive 8mb of memory (that's the size of the csv), maybe twice, I dunno, but I do know it was an easy fix that scales well.

The controllers

The real memory hog is the array we create in the controllers, but how else can we have the code in the controllers, but have the data in the view? Callbacks. Now bear with me, it's not a dirty word as it sounds, not if done correctly.

Conclusion

so now we have nice neat code in the controller that's going to be called in the view, and it's passed a closure (i might have used the wrong terms through this, but as long as you got the point who cares? :) ) that will convert the hash passed to it into a csv row, and output it to the response.

now, we can run the action like before, but memory usage stays nice and low, and it's neater imo, although it's also more complex.

Most of the time you won't need to do this, i've not had to in 6 years of Catalysting, but when I did, it worked a treat.

p.s. the above code might not compile, it's half pulled from git logs and memory, and hasn't been tested. but the idea is sound, it's been live on our site for months now no troubles.