On Wed, Jan 7, 2009 at 8:00 PM, Brian Granger <ellisonbg.net@gmail.com> wrote:
> Hi,
>> I see that people are starting to use multiprocessing to parallelize
> numerical Python code. I am wondering if we want to allow/recommend
> using multiprocessing in scipy. Here are some of my concerns:
>> * Currently multiprocessing doesn't play well with IPython. Thus, if
> scipy starts to use multiprocessing, people will get very unpleasant
> surprises when using IPython. I don't know exactly what the problems
> are, but my feeling is that it is unlikely that IPython will ever have
> *full* support for multiprocessing. Some support might be possible,
> though.
I've used multiprocessing (or actually pyprocessing) a little bit with
IPython. The main problem is that you can't use interactively defined
functions - ie can't do something like
p = Pool(8)
p.map(lambda x: somefunc(x,2,3), range(1,100))
because pickling interactively defined stuff doesn't work in Ipython.
Other than that though - it's been working fine for me, (ie just make
sure anything you are using is defined in a module so pickle works):
from module import somefunc
p.map(somefunc, range(1,100)
although I haven't been trying to do anything too clever (haven't had
any plots open or anything like that)
I think it adds a very valuable feature - that for a beginner like me
is much easier to get to grips with than MPI or even the clustering
features of ipython, to easily allow use of multi-core machines.
It would be great if IPython could sort out the pickle business so you
could pickle interactively defined functions (they currently don't
show up in __main__ which is a FakeModule instance).
Robin