Project description

Wendelin.core allows you to work with arrays bigger than RAM and local disk.
Bigarrays are persisted to storage, and can be changed in transactional manner.

In other words bigarrays are something like numpy.memmap for numpy.ndarray
and OS files, but support transactions and files bigger than disk. The whole
bigarray cannot generally be used as a drop-in replacement for numpy arrays,
but bigarray slices are real ndarrays and can be used everywhere ndarray can
be used, including in C/Cython/Fortran code. Slice size is limited by
virtual address-space size, which is ~ max 127TB on Linux/amd64.

The main class to work with is ZBigArray and is used like ndarray from
NumPy:

we are currently working on improved wendelin.core design and implementation,
which will use kernel virtual memory manager (instead of one implementedinuserspace) with arrays backend presented to kernel via FUSE as virtual
filesystem implemented in Go.

Wendelin.core change history

0.12 (2018-04-16)

Add ArrayRef utility to find out for a NumPy array its top-level root
parent and how to recreate the array as some view of the root;
this builds the foundation for e.g. sending arrays as references without copy
in CMFActivity joblib backend
(commit 1, 2, 3).

Don’t crash if during loadblk() garbage collection was run twice at tricky
times (commit 1, 2).