I am interesting in using sqlite (or pytables) to store data for scientific
research. I wrote the attached test program to save and load a simulated
11x500,000 recarray. Average save and load times are given below (timeit
with 20 repetitions). The save time for sqlite is not really fair because I
have to delete the data table each time before I create the new one. It is
still pretty slow in comparison. Loading the recarray from sqlite is
significantly slower than pytables or cPickle. I am hoping there may be more
efficient ways to save and load recarray¹s from/to sqlite than what I am now
doing. Note that I infer the variable names and types from the data rather
than specifying them manually.
I¹d luv to hear from people using sqlite, pytables, and cPickle about their
experiences.
saving recarray with cPickle: 1.448568 sec/pass
saving recarray with pytable: 3.437228 sec/pass
saving recarray with sqlite: 193.286204 sec/pass
loading recarray using cPickle: 0.471365 sec/pass
loading recarray with pytable: 0.692838 sec/pass
loading recarray with sqlite: 15.977018 sec/pass
Best,
Vincent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_tables_test.py
Type: application/octet-stream
Size: 4154 bytes
Desc: not available
Url : http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.obj