Hi there, i have write a script that manages input files i give. It creates threads to make the whole process faster. The operation takes about 5 minutes with threads while doing it without theads it takes almost 40 minutes. The time is of course analog to the filesize and the number of the threads. For a file around 4.7Megabyte i create 408 threads which run for about 5 minutes and in that time the cpu is 100% working as i see in Task Manager on win7. When the threads finish their jobs i see that the cpu goes down at 15% (at task manager). But, the perl.exe process doesn't release the memory which has reached around 400megabyte and i have to close the program to release the memory. I tried, after the finish of threads, many ways to release the memory such as

Code

for($a=0;$a<=$filecounter;$a++){ $thr[$a]->exit(); }

and other like

Code

for($a=0;$a<=$filecounter;$a++){ $thr[$a]=undef; }

or

Code

for($a=0;$a<=$filecounter;$a++){ die $thr[$a]; }

but i can't release the memory back to the os. I only manage to shut down the whole program with the ways i wrote above or take some warning messages. I would appreciate the help and the opinion whoever can help me. Thank you :)

I used semaphores th shut down the threads but then the memory in use just get doubled and reached 800Megabyte. I closed the whole process because soon it would get out of memory. I have read the doc on perldoc page for the threads but really can't find the way out.

The loop really didn't help much. I created an example basing it on your loop contents.

Do you have loops that check if the threads are ok to join (is_joinable) and ok to exit (is_running)?

About join, this means block or wait until the thread is complete and then release the thread and do any OS cleanup thats needed. Its important to join. Detach means to ignore the thread, which will just exit and release when its done.

I ran this on linux, adding the sleep after everything was done, allowed the system time to free memory. In this run, it freed up ~80KB.

I will check the code you posted and i will come back soon to reply. As you see, i have a while loop which won't stop until the $sum_a isn't zero (when it becomes zero then all threads have finished their job). Anyway, thank you very much for your interest. I will reply soon to your code.

EDITAs you see, i don't know what to do after the block of code i posted above where i am pretty sure that the threads have finished their jobs but the memory remains. I really don't know how to release the memory and i have tryed the options in my first post. All the options i have tryed from my first post have been tested after the block of code i posted here in my third post. I would really appreciate it if you could help me somehow with my problem.

To use threads you need to do these steps: 1) create the threads (OK) 2) wait til they can be joined (you need to do this) 3) join the threads (you need to do this - this initiates the cleanup when done) 4) check that all threads exited (OK)

You need to do steps 2 and 3, otherwise, the threads remain open.

There are two ways to close a thread, join or detach. (You aren't doing either)

If you care about the output, and want to retrieve it, then you join each thread when they able to join. You must check if they can join, otherwise the parent can send the signal, and the child never sees it.

If you are not concerned about the output, then you can use detach, which tells the thread to cleanup when you are done, the parent doesn't care.

Either way, the memory cleanup is handled by join and detach. When it's done depends on which one is used.

The program doesn't release the memory and when i shut it down i get the below info (i put it just for information, as i had output messages like that and before) Perl exited with active threads: 0 running and unjoined 2 finished and unjoined 0 running and detached

Sorry but I really don't know where to put the things you told me. I don't care about the output of the threads since they finish their jobs and as you told me I used the detach choice but I guess I have more errors in my code. :(

Let me clarify a bit more about join and detach, I never really used detach, my understanding was it let the processes complete when done, that was incorrect, any processes that are detached will be killed silently when parent quits.

Joinable means you want to capture data from the threads. You use is_joinable to allow you to do other things, while the thread is running. If you just issued the join without checking, the loop would wait (block) until the thread completed.

Detach means you don't care about the output of the thread, you are done with it. Once detached, you cannot rejoin it. You monitor it by checking is_running and if detach was successful with is_detached.

I managed to detach all the finished threads but the memory remains Anyway, i spent free time from a week to achieve the memory release but found nothing. I also tried fork() but I have issues over there too I can't found anything else on net helping me. The only I have do - but can't let it like that for sure - is to restart the program when the threads finish. But, this is only for now, I can't leave it like that. If you have any more ideas for a possible solution you just can post it in the thread. Anyway, no matter the result, i really thank you for your interest and your willing to help me

I did find some notes about perl threads under Windows, they don't always work the best. Perl uses its own threads, and from what I read, each thread launches is own Perl environment, so the overhead can be quite high if you are loading up 400+ threads.

To see if there is another possible solution, I need to know more about what is going on with ext1 reset and resume. Also, how the files are processed, or fed into the script. Is this a resident app like a daemon or tsr?

Because of the way perl is processing, you may need to treat these files as sessions, so the perl.exe exec can exit and do cleanup. Its hard to tell, I would definitely need more details, even a brief outline of the process so I can see how files are spawned into threads.