An Update on MarFS in Production

In this video from the MSST 2017 Mass Storage Conference, David Bonnie from LANL presents: An Update on MarFS in Production. MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. Extreme HPC environments require that MarFS scale a POSIX namespace metadata to trillions of files and billions of files in a single directory while storing the data in efficient massively parallel ways in industry standard erasure protected cloud style object stores.

“With MarFS in production at LANL since fall 2016, we have gained new insights, learned lessons, and expanded our future plans. We’ll discuss the various hurdles required to deploy such an ambitious system with minimal manpower. Further, we’ll delve into the challenges, triumphs, and defeats on the road to a new tier of inexpensive scalable storage.”

Many computing sites need long-term retention of mostly cold data often “data lakes”. The main function of this storage tier is capacity but non trivial bandwidth/access requirements exist. For many years, tape was the best economic solution. Data sets have grown larger more quickly than tape bandwidth improvements and access demands have increased in the HPC environment. Disk can be more economically for this storage tier. The Cloud Community has moved towards erasure based object stores to gain scalability and durability using commodity hardware. The Object Interface works for new applications but legacy applications utilize POSIX for their interface.

David Bonnie is a scientist with a background in scalable storage systems and software development. He contributed to the development of OrangeFS/PVFS2 and holds a deep interest in understanding current and future storage hardware and software. He is currently the campaign storage and enterprise backup technical lead for the HPC Division at Los Alamos National Laboratory, both positions that include leading integration, development, and production support efforts for LANL’s pre-archive and backup tiers. He is integral in the development and deployment of MarFS, LANL’s new filesystem designed to serve the needs of ever-growing datasets with spiraling bandwidth and reliability challenges. David holds a B.S. and M.S. in Computer Engineering, both from Clemson University.

Resource Links:

Latest Video

Industry Perspectives

Addison Snell gave this talk at the Stanford HPC Conference. “Intersect360 Research returns with an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2017 and beyond. Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations.” [READ MORE…]

White Papers

XTREME-D’s next step in offering seamless HPC/DA/DL computing is the XTREME-Stargate gateway platform, or cluster portal. This on-prem device acts as a cluster “head node” and provides secure and fast access to bare-metal clusters that are configured using enhanced XTREME-DNA technology. Download the full paper to learn more about how XTREME-Stargate offers a “super head node” for HPC cloud clusters.