Francesc Alted wrote:
> A Wednesday 28 July 2010 18:05:11 脑关(BrainGateway)生命科学仪器 escrigué:
>> I had some 500GB numerical arrays
>> to process. Pytable claims to have some advance feature to enhance
>> processing speed and largely reduce physical memory requirement. However, I
>> do not wanna touch the raw data I had. Simply because I do not have doubled
>> diskspace to covert all 10TB data into HDF5. Is there any way to let
>> pytable read raw binary files or alternatively to package raw files into
>> HDF5 format without change the files themselves.?
>> No, PyTables does not support raw data directly. You have to convert to HDF5
> first.
Which is probably worth it.
But if your data files are pretty simple, you may be able to use a
memory mapped array, or baring that, write your own reader with fromfile().
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov