Okay, thanks for the input, all. I've just fiddled around with it a
bit, and it seems that Robin's suggestion makes the most sense.
Define a module that has the persistent data in it. That can then
provide load / unload type functions that manage the data it stores.
Used inside of ipython it looks like it prevents needing to re-load
data unnecessarily.
As an aside, it looks to me like the cPickle stuff is actually
substantially faster than I thought it might be, so it's probably not
as prohibitive as I thought to use that as a session 'scratch file' or
whatever one wants to call it.
Now it's just a matter of converting ~13k lines of IDL into python :D
Cheers,
Dave
On Fri, Jul 23, 2010 at 8:01 AM, Sebastian Haase <seb.haase@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 2:26 AM, Charles R Harris
> <charlesr.harris@gmail.com> wrote:
>>>>>> On Wed, Jul 21, 2010 at 2:18 AM, David Andrews <irbdavid@gmail.com> wrote:
>>>>>> Hi All,
>>>>>> I suppose this might not strictly be a scipy type question, but I'll
>>> ask here as I expect some of you might understand what I'm getting at!
>>>>>> I'm in the process of porting some code from IDL (Interactive Data
>>> Language - popular in some fields of science, but largely nowhere
>>> else) to Python. Essentially it's just plotting and analyzing time
>>> series data, and so most of the porting is relatively simple. The one
>>> stumbling block - is there an equivalent or useful replacement for the
>>> "common block" concept in IDL available in Python?
>>>>>> Common blocks are areas of shared memory held by IDL that can be
>>> accessed easily from within sub-routines. So for example, in our IDL
>>> code, we load data into these common blocks at the start of a session,
>>> and then perform whatever analysis on it. In this manner, we do not
>>> have to continually re-load data every time we re-perform a piece of
>>> analysis. They store their contents persistently, for the duration of
>>> the IDL session. It's all for academic research purposes, so it's
>>> very much 'try this / see what happens / alter it, try again' kind of
>>> work. The loading and initial processing of data is fairly time
>>> intensive, so having to reload at each step is a bit frustrating and
>>> not very productive.
>>>>>> So, does anyone have any suggestions as to the best way to go about
>>> porting this sort of behavior? Pickle seems to be one option, but
>>> that would involve read/write to disk operations anyway? Any others?
>>>>>>> Depending on the sort of data you have, PyTables might be an option. I'm
>> currently using it to store a 42 GB image data cube on disk and it works
>> well for that. I can browse through an image and shift-click on a pixel to
>> get a plot of the data associated with the pixel. It is quite fast. The data
>> cube needs to be passed as an argument to the various functions that need
>> the data, but that isn't much of a problem.
>>> Chuck, just out of curiosity: what are the specs of your hardware and
> which OS are you on ?
>> -Sebastian Haase
> _______________________________________________
> SciPy-User mailing list
>SciPy-User@scipy.org>http://mail.scipy.org/mailman/listinfo/scipy-user>