STXXL

The core of STXXL is an implementation of the C++ standard template library STL for external memory (out-of-core) computations, i. e., STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks. While the closeness to the STL supports ease of use and compatibility with existing applications, another design priority is high performance.

Small constant factors in I/O volume. A unique library feature called "pipelining" can save more than half the number of I/Os, by streaming data between algorithmic components, instead of temporarily storing them on disk. A development branch supports asynchronous execution of the algorithmic components, enabling high-level task parallelism.

Shorter development times due to well known STL-compatible interfaces for external memory algorithms and data structures.

STL algorithms can be directly applied to STXXL containers; moreover, the I/O complexity of the algorithms remains optimal in most of the cases. [more info]

For internal computation, parallel algorithms from the MCSTL or the libstdc++ parallel mode are optionally utilized, making the algorithms inherently benefit from multi-core parallelism.

Contains memory management, disk virtualization, prefetching, and so on, as the lower layers, and as part of the higher layer (pipelined) sorting with SMP and multi-core processor support, (pipelined) scanning and containers (vectors, stacks, priority queues, maps (B+Tree), queues, deques). Currently that sums to about 35,000 lines of code.

Branches

Special features are maintained as Github forks until they are merged into master. Until inclusion into the master branch, the interface may change without further notice.

Asynchronous Pipelining/Streaming: parallel_pipelining_integrationThis contains an unfinished integration attempt of the async nodes and parallel sorting/pipelining described in the IPDPS 2009 paper. If someone has a stake or interest in this branch, please contact me (Timo) about further work on it. -- 2014-10-23

Publications, Ongoing and Completed Projects using STXXL

If you use STXXL and wish to appear in the following list, please provide a description line via email to one of the maintainers.