Threadsafety and reusing PythonInterpreter or locals / state which was initialized previously?

Heimo Laukkanen <huima <at> iki.fi>
2010-02-26 17:22:52 GMT

Hi all,
I am integrating Jython into java web-application and was wondering
about the possibility to reuse PythonInterpreter's state after initial
setup.
Jython is used as a calculation engine, in which calculation consists
of files that define constants used in calculations and functions (
methods ) that take variables from users and returns results in the
end.
What I would like to do, is to be able to initialize PythonInterpreter
once, load all the code with all the imports and then make copies of
PythonInterpreter with that state for each calculation. Reason for
this is that initial setup - loading all the imports etc. takes about
4000 milliseconds on my slow laptop, where as individual calculations
and even restoring the interpreter state take just few milliseconds.
I found similar question asked in 2001:
http://aspn.activestate.com/ASPN/Mail/Message/Jython-users/468265
PyStringMap localsMap = (PyStringMap) interpreter.getLocals();
PyObject initialState = (PyObject) localsMap.copy();
The thing I was left wondering was what happens with variables defined
inside function and whether there will be thread safety issues as copy
from locals is just a shallow copy and objects point to same objects.
If that is the case, would it be sufficient to put all the code inside
classes and instantiate new object for each instance of calculation

Re: Threadsafety and reusing PythonInterpreter or locals / state which was initialized previously?

Nicholas Riley <njriley <at> illinois.edu>
2010-02-27 05:42:01 GMT

In article
<ac4b62c71002260922x42fe10a1t8b0756b14fd07da4 <at> mail.gmail.com>,
Heimo Laukkanen <huima <at> iki.fi> wrote:
> I can and could do a testcase launching numerous threads to try it
> out, but would be more comfortable to get an answer from someone who
> actually knows Jython internals.
Jython's internals itself should be fine to use from multiple threads.
There's no way to isolate arbitrary behavior, but if you restrict
yourself to reading and writing from local variables, then it should be
just fine.
Last month I added a thread local option to PythonInterpreter, which you
might find useful:
/**
* Creates a new interpreter with the ability to maintain a
* separate local namespace for each thread (set by invoking
* setLocals()).
*
* <at> param dict
* a Python mapping object (e.g., a dictionary) for use
* as the default namespace
*/
public static PythonInterpreter threadLocalStateInterpreter(PyObject
dict)
Using a separate PySystemState for each thread would also be another
option (which I've actually implemented at some point), which would

Re: Threadsafety and reusing PythonInterpreter or locals / state which was initialized previously?

Heimo Laukkanen <huima <at> iki.fi>
2010-02-27 11:25:00 GMT

>Message: 6
>Date: Fri, 26 Feb 2010 23:42:01 -0600
>From: Nicholas Riley <njriley <at> illinois.edu>
>Message-ID: <njriley-5495A7.23420026022010 <at> news.gmane.org>
>
>Jython's internals itself should be fine to use from multiple threads.
>There's no way to isolate arbitrary behavior, but if you restrict
>yourself to reading and writing from local variables, then it should be
>just fine.
Excellent.
I tried to check out from Jython sourcecode how everything works, but
was not 100% sure as I had not read and reasoned everything through.
But based on how Python works, what you wrote and what I saw in the
source code, it seems safe to assume that for each function invocation
new local namespace ( public PyObject __dict__ in PyFunction in Jython
) is created and therefore even though functions themselves would be
defined in global namespace and shared between interpreters there
should not be any problems.
Defining variables for calculation and setting result from function
call would be set in new interpreter's local namespace, which is
shallow copy from the original - pre instantiated namespace. These new
names would therefore be local only to that instance.
>
>Last month I added a thread local option to PythonInterpreter, which you
>might find useful:
>

Re: Threadsafety and reusing PythonInterpreter or locals / state which was initialized previously?

Nicholas Riley <njriley <at> illinois.edu>
2010-02-27 20:36:30 GMT

In article
<ac4b62c71002270325p55a62529t7d467d734f2537a3 <at> mail.gmail.com>,
Heimo Laukkanen <huima <at> iki.fi> wrote:
> Any information on when that code from SVN will be included in a
> release, and distributed through some maven repository?
Hopefully in a few weeks - we do plan on doing a 2.5.2 release ASAP.
--
--
Nicholas Riley <njriley <at> illinois.edu>
------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev