14.05.17 - 19.05.17, Seminar 17202

Challenges and Opportunities of User-Level File Systems for HPC

The following text appeared on our web pages prior to the seminar, and was included as part of the invitation.

Motivation

Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention.

Many such user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, it would benefit the larger community if we could develop a common framework for HPC file system development. The framework we envision would be based on solid software engineering practices and would enable “plug and play” interchange of basic file system components. Having this framework would facilitate both the operation needs of current production systems and the research efforts for novel features in future systems. On the production side, the framework would provide a trusted backbone upon which users and developers could rely for basic services. For research, the backbone of the framework would enable rapid prototyping of file system components so that students and other researchers did not have to implement the entire framework simply to experiment with a specific optimization.

The goal of this Dagstuhl Seminar is to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in designing a user-level file system framework for HPC systems. We expect a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file system framework. Our hope is that the effort initiated during this seminar will result in a long-term collaboration that will benefit the HPC community as a whole.