This is the sixth in a series of articles on “6 Things You Should Know About Lustre”. See links below to find other topics covering Lustre in Enterprise, the Cloud, Financial Services, and next-generation storage.

With terabytes to petabytes of data to move—in massively large single-file data sets or millions of small files—users of today’s High Performance Computing (HPC) systems are more concerned about the efficiency and speed of data movement than ever before. Moore’s Law has allowed processor designers and manufacturers over the years to take the bottlenecks out of compute, moving the choke points to other areas of the system—storage, memory, and communications (fabric). HPC integrators and operators have battled for years with building balance into these massive, complex systems to optimize the overall performance.

The recently revealed Intel® Scalable System Framework (Intel® SSF) is an approach to enable builders and operators to more easily architect and integrate flexible, performant, and balanced HPC solutions for their customers. Intel has assembled the best of breed components, from Intel-enhanced, open source community-based software to power-efficient and high-performance hardware technologies of their own, into a framework on which next-generation HPC is being constructed by leading OEMs and integrators in the industry.

Lustre, proven for its throughput in research labs for years, is now among the top few, leading parallel file systems capable of meeting the needs of the HPC community—both in research and enterprise. Some of the most recent HPC installations with Lustre are seeing hundreds of gigabytes[1] to terabytes[2] per second of throughput. The company has incorporated Intel® Solutions for Lustre Software as part of the Intel SSF because it provides the performance to move data and minimize storage bottlenecks. Lustre is also open source based, and already enjoys a wide foundation of deployments in research around the world, while gaining significant traction in enterprise HPC. Intel’s version of Lustre delivers a high-performance storage solution in the Intel SSF that next-generation HPC needs to move toward the era of Exascale.

Performance alone, however, is not enough to become part of a total solution HPC framework in today’s IT environments. Reliability, manageability, and flexibility are top of mind for enterprise HPC, where Total Cost of Ownership is right up there in importance with performance.

Today’s Lustre has undergone many years of evolution by contributions from the developer community and additional tools added by Intel engineers for the Intel versions to make it best of breed for not only traditional HPC workloads, but for visualization, machine learning, and Big Data Analytics. The mature features these enhancements provide, such as High Availability (HA), Hierarchical Storage Management (HSM), reliability, and adaptability for Hadoop, are now part of core Lustre capabilities. Plus, Intel has added the Intel® Manager for Lustre in their own enterprise version, which simplifies and accelerates Lustre deployment and management. Innovative security and replication features are among the forefront of continuing Lustre enhancements.

Key to the goals for Intel SSF is to deliver balance and move data closer to the compute cores. CPU manufacturers have proven over the years that getting data closer to the cores improves overall efficiency and performance. That’s why we’re seeing more and more integration into the processor silicon of functions that used to be outboard, such as memory and graphics. Intel is taking it a step further and integrating part of the fabric into the processor to accelerate data movement to remove communications bottlenecks. Intel SSF is also working on innovations in memory and SSDs, integrated at the system level, all to help move more data to the compute complex faster.

But data begins in the file system, which is why Lustre and Intel Solutions for Lustre software, are critical to enabling Intel’s new framework for next-generation HPC solutions.

Resource Links:

Latest Video

Industry Perspectives

In this podcast, Terri Quinn from LLNL provides an update on Hardware and Integration (HI) at the Exascale Computing Project. "The US Department of Energy (DOE) national laboratories will acquire, install, and operate the nation’s first exascale-class systems. ECP is responsible for assisting with applications and software and accelerating the research and development of critical commercial exascale system hardware. ECP’s Hardware and Integration research focus area (HI), was created to help the laboratories and the ECP teams achieve success through mutually beneficial collaborations." [Read More...]

White Papers

For this report, DDN performed a number of experimental benchmarks to attain optimal IO rates for Paradigm Echos application workloads. It present results from IO intensive Echos micro-benchmarks to illustrate the DDN GRIDScaler performance benefits and provide some detail to aid optimal job packing in 40G Ethernet clusters. To find out the results download this guide.