David Bonnie from LANL presented this talk at the MSST 2017 Mass Storage Conference. MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. “With MarFS in production at LANL since fall 2016, we have gained new insights, learned lessons, and expanded our future plans. We’ll discuss the various hurdles required to deploy such an ambitious system with minimal manpower. Further, we’ll delve into the challenges, triumphs, and defeats on the road to a new tier of inexpensive scalable storage.”

Gary Grider from LANL presented this talk at the Storage Developer Conference. “MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. Extreme HPC environments require that MarFS scale a POSIX namespace metadata to trillions of files and billions of files in a single directory while storing the data in efficient massively parallel ways in industry standard erasure protected cloud style object stores.”

David Bonnie from LANL presented this talk at the 2016 MSST Conference. “As we continue to scale system memory footprint, it becomes more and more challenging to scale the long-term storage systems with it. Scaling tape access for bandwidth becomes increasingly challenging and expensive when single files are in the many terabytes to petabyte range. Object-based scale out systems can handle the bandwidth requirements we have, but are also not ideal to store very large files as objects. MarFS sidesteps this while still leveraging the large pool of object storage systems already in existence by striping large files across many objects.”

“We wanted to get away from the complexity of POSIX for data, yet retain the parts of POSIX that people are used to (metadata manipulation). By divorcing ourselves from the complications of ensuring a completely POSIX data flow, we can massively simplify the data movement and storage mechanisms. MarFS lets us retain the parts of POSIX that users appreciate for data management (chown, chmod, rename, mv, etc) without inheriting the complexity of managing POSIX semantics for data manipulation. By treating the data as essentially immutable, we can leverage the very simple PUT/GET/DELETE semantics of “cloudy” data storage systems to scale out storage with ease.”

“Trends in computer memory/storage technology are in flux perhaps more so now than in the last two decades. Economic analysis of HPC storage hierarchies has led to new tiers of storage being added to the next fleet of supercomputers including Burst Buffers or In-System Solid State Storage and Campaign Storage. This talk will cover the background that brought us these new storage tiers and postulate what the economic crystal ball looks like for the coming decade. Further it will suggest methods of leveraging HPC workflow studies to inform the continued evolution of the HPC storage hierarchy.”

Latest Video

Industry Perspectives

Often, it’s not enough to parallelize and vectorize an application to get the best performance. You also need to take a deep dive into how the application is accessing memory to find and eliminate bottlenecks in the code that could ultimately be limiting performance. Intel Advisor, a component of both Intel Parallel Studio XE and Intel System Studio, can help you identify and diagnose memory performance issues, and suggest strategies to improve the efficiency of your code. [READ MORE…]

White Papers

This pioneering study focuses primarily on the virtual performance of throughput workloads. Download the new white paper from VMWare that explores the possibilities of virtualizing HPC throughput in computing environments.