multicpu bzip2 using os.system or queue using python script

I want to quickly bzip2 compress several hundred gigabytes of data
using my 8 core , 16 GB ram workstation.
Currently I am using a simple python script to compress a whole
directory tree using bzip2 and a system call coupled to an os.walk
call.

I see that the bzip2 only uses a single cpu while the other cpus
remain relatively idle.

I am a newbie in queue and threaded processes . But I am wondering how
I can implement this such that I can have four bzip2 running threads
(actually I guess os.system threads ), each using probably their own
cpu , that deplete files from a queue as they bzip them.

Advertisements

harijay wrote:
> I want to quickly bzip2 compress several hundred gigabytes of data
> using my 8 core , 16 GB ram workstation.
> Currently I am using a simple python script to compress a whole
> directory tree using bzip2 and a system call coupled to an os.walk
> call.
>
> I see that the bzip2 only uses a single cpu while the other cpus
> remain relatively idle.
>
> I am a newbie in queue and threaded processes . But I am wondering how
> I can implement this such that I can have four bzip2 running threads
> (actually I guess os.system threads ), each using probably their own
> cpu , that deplete files from a queue as they bzip them.
>
>
> Thanks for your suggestions in advance
>
[snip]
Try this:

Share This Page

Welcome to The Coding Forums!

Welcome to the Coding Forums, the place to chat about anything related to programming and coding languages.

Please join our friendly community by clicking the button below - it only takes a few seconds and is totally free. You'll be able to ask questions about coding or chat with the community and help others.
Sign up now!