jdlev has asked for the
wisdom of the Perl Monks concerning the following question:

I'm not sure I understand this. I'm running a perl program that runs through a couple billion loops looking for an optimal scenario. It shouldn't be storing anything long term? When I run the program, it gets to a certain point and seems to stop...I'll hit control-c to stop the program and it says "Out of memory!" I'm not sure how that's even possible since my computer has almost 60GB of memory installed?
Does perl have any type of error reporting or log writing abilities so I can at least see when then program is crashing?

My OCD kicked in and while I could not find any particular memory-related errors, I 'fixed' the program's indentation so that it should be more legible (so the other monks have an easier time looking at it).

A couple of more obvious errors were fixed, too. Please check that the code still behaves the same.

You really ought to use warnings at the very least; it might provide pertinent clues to the issue.

You have "foreach" loops 10 deep in that program. If each one runs only 3 times, that's 60,000 iterations of the inner loop. I would seriously consider rethinking what you're trying to do, or how you're doing it, because an O(n^10) algorithm is usually a recipe for disaster.

You have several identical cases of sort keys $someHash within nested loops. If the list of keys is big, especially very big, that could consume a lot of memory. Yet I do not superficially see anything within the accompany logic that mandates the sort in order to produce correct results.

It occurs to me that much of this logic appears to be similar to what might be obtained ... at least, in part ... by an SQL INNER JOIN query. You’ve got a lot of nested loops here, none of which seem to be doing much more than to set-up for the next inner nested loop. And, well, “if it’s combinations that you want, even with ORDER BY, then SQL is already fantastic at doing that sort of thing. How much of this “nested loop” logic might be replaceable by a single SQL query, involving many JOINs, which would present all of those combinations to you naturally as part of its result?

Pursuing the last thought, I frankly do suspect that a lot of this “nested loop” logic could, indeed be expressed as a query which might, indeed, produce many thousands of rows as a “cross-product” between several smaller constituent tables. (“And so what... that’s what SQL servers do for a living...”) But this might then serve to rather-drastically reduce the complexity and memory-footprint of your code, which now only has to consume a record-set that is presented to it.

Note also that “SQL” doesn’t have to imply “a server.” The SQLite database system, for instance, is built on single-files, and it runs quite nicely on everything from mainframes to cell-phones. My essential notion here is that maybe you can shove “all that data” out of (virtual...) memory, and into file(s).

Yes, it is also not clear to me whether sorting the hash keys is necessary, but if it is necessary, you might consider sorting the keys before entering the nested loops and storing them into an array (and walking through the arrays containing the sorted keys rather than the hash keys). I do not know whether it will really reduce memory usage sufficiently, but it will certainly reduce considerably the run time. The hash keys used in the most inner loops might be sorted millions or possibly even billions of times, this is a huge waste of CPU power, as your program is likely to spend most of its running time sorting again and again the same data. Now, to restate, this will certainly also save some memory, but I have no idea whether this will be enough to solve your memory problem.

One additional point: how many elements do you have in each of your hashes?

I made the following changes: presorting the hash keys into arrays and using the arrays of keys to walk though the data, added my statements to declare your variables so that you can enable warnings and strictures (I might have missed some, I can't really test or compile).

Three additional points: (1) I do not see the point of using a delay in your most inner loop, I think you can remove it. (2) The fact that you did not give lexical scope to your $key loop variable is probably a major bug in your program, because I think that the $key variable at one level of nesting gets overwritten by the last value taken by $key within the next nested loop (not entirely sure though, I have not tried in ages to use such loop variable without lexically scoping it, maybe Perl can manage this correctly, but I would be surprised). OTOH, it may have no impact because starting the inner loop is the last thing you do each time, but this is poor and dangerous design. Anyway, this should no be the case with my version above, where each $key variable is lexically scoped within its loop. (But, again, I can't test anything.) (3) Finally, the $salary variable used at the end of the program seems to be coming from nowhere. Using warning and strictures would detect this, just as the fact that you are using $maxPoint and $maxPoints.

When putting a smiley right before a closing parenthesis, do you:

Use two parentheses: (Like this: :) )
Use one parenthesis: (Like this: :)
Reverse direction of the smiley: (Like this: (: )
Use angle/square brackets instead of parentheses
Use C-style commenting to set the smiley off from the closing parenthesis
Make the smiley a dunce: (:>
I disapprove of emoticons
Other