guice has asked for the
wisdom of the Perl Monks concerning the following question:

Anybody have any problems with Parallel::ForkManager or maybe even DBI eating up resources?

I currently run a script that forks upto 5 children using ForkManager with upto a total 140 children by the time it's done.

When I run a big load (133), my CPU load slowly climbs from 5 to over 20 by the time it finishes. Once the whole script is done, it drops back down to ~5-7ish (not immediately, but 2x the speed it climbed).

Any thoughts?

I do everything by the book; Opening a DBI process per child, closing and unsetting the veriable for each close.

Ouch. 140 children all with their own DB connections? And you create a new connection per child? Surely there is a better way.

First off, you should probably cut down the number of children. Most database performance numbers fall off pretty quickly when you start adding a number of children doing anything more than reads.

You should also look at using a pool of database handles. Why create and then close database connections over and over again if you don't have to? I sped up the performance of a data loading script by a few orders of magnitude by moving the dbh create/destroy outside of the main loop.

It's a max of 5 children at once. I tried to use a pool, but couldn't figure out a way to make sure none of the children used a wrong handle. Care to share how you got past this one? I tried an array with $cnt++ % 5, but no way to guarentee each child will finish in order.

Each child starts off opening a connection and then calls a close method which closes the connection. All that works and in order. The code is very verbose and I see that part just fine.

Now I'm at a loss, how did you successfully create a pool of DBI handles with multiple children? I have a pool set, but as soon as my first child ends, DBI tries to clean itself up and kills all my other DBI handles with "end-of-file on communication channel during global destruction" error.

Without seeing your code, it is hard to tell. But it sounds like you might not be waiting for your current children to die before spawning new children. Then finally at the end of the parent program, all the children are reaped.

Why are you creating & closing oracle connections? Unless there is some special reason to do that (aka, tasks running as different oracle users), don't do that. Creating & destroying oracle connections so often is where your load jump is coming from.
You should switch your model from:

You're suggesting taking the pool of tasks, split them into 5 arrays and then run them?

My tasks is updating a load of 100+ server stats in a database. I'll have to use some kind of splitting to split the array of hosts into 5 arrays (as evenly as possible) for splitting into individual forks.