2013/1/29 Johann Rohwer <jr@sun.ac.za>
> On Tuesday 29 January 2013 13:37:34 Florian M. Wagner wrote:
> > I am currently trying to parallelize a piece of code using IPython's
> > parallel version of map. It works fine for simple examples, but the
> > function I would like to distribute falls back upon some globally
> > declared variables, which the engines are not aware of. Is there a
> > way to sync the namespace of all engines analogously to
> > sync_imports or what would you suggest?
>> I'm not aware of whether it is possible to sync the namespaces
> completely and whether this useful or feasible at all (do we want the
> controller and client objects to be available on all the engines?).
> However, you can easily push those global variables the function
> requires to all the engines with the dview.push() method.
>> Johann
> E-pos vrywaringsklousule
>> Hierdie e-pos mag vertroulike inligting bevat en mag regtens
> geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit
> geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee
> in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of
> kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in
> kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid
> vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of
> die oopmaak van enige lês aangeheg by hierdie e-pos nie.
>> E-mail disclaimer
>> This e-mail may contain confidential information and may be legally
> privileged and is intended only for the person to whom it is addressed. If
> you are not the intended recipient, you are notified that you may not use,
> distribute or copy this document in any manner whatsoever. Kindly also
> notify the sender immediately by telephone, and delete the e-mail. The
> University does not accept liability for any damage, loss or expense
> arising from this e-mail and/or accessing any files attached to this e-mail.
>> _______________________________________________
> IPython-User mailing list
>IPython-User@scipy.org>http://mail.scipy.org/mailman/listinfo/ipython-user>>Hi Florian,
I asked something related a few month ago: my main problem was to `import
module as m` and broadcast functions and variables to all the engines. As
now that is no automatic way.
So I wrote a class that initialize my parallel environment, and perform a
few operations. One of the methods is
def exec_on_engine( self, code, block=True ):
"""
Execute the given code on all engines
Parameters
----------
to_execute: string or list of strings
command(s) to execute on all the nodes. Thought for short tasks,
like importing modules
block: bool
whether or not to wait until done to return. default: True
"""
if isinstance(code, basestring): # if it's a string
code = [code,] # convert to list
# execute the required commands
# (better to do in block mode, avoids errors if command is slow)
for te in code:
try:
self.dview.execute(te, block=block)
except error.CompositeError, e: # if an error occurs, print a
single one, not one per engine
e.raise_exception()
I use it roughly as follow (I didn't test passing a variable, but should
work)
import os
# absolute path and file name of this script
path, fname = os.path.split(os.path.abspath(sys.argv[0]))
#command to run on all the engines
imports = ['import numpy as np',
# add the script directory to the python path
'import sys', 'sys.path.append("{0}")'.format(path),
# import a function from the script in the namespace
'from {0} import func'.format(os.path.splitext(fname)[0]),
'a = 10']
parallel_env.exec_on_engine(imports)
Does it help?
Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130129/40c2814e/attachment.html