OVERVIEW: Participating in the HPC Storage Challenge

Teams participating in the SC09 HPC Storage Challenge will begin work at their earliest convenience and must submit a Project Proposal by July 27, 2009; the Project Proposal is the Submission. The Project Proposals will be used to select a small number of finalist teams who will be notified of their status by August 17, 2009.Finalist teams may continue working on their challenge projects after submitting their Project Proposal until the Final Project Report is due (November 9, 2009). During the conference, finalist teams will deliver a technical session presentation that will be part of the Storage Challenge judging. They will also participate in a private follow up interview with the committee. Titles, abstracts and team member names will be posted in the Conference Program and the winners will be announced at the conference awards session.

HPC Storage Challenge Resources

Teams participating in the SC09 HPC Storage Challenge will work on their challenge projects in advance of the conference with resources they can obtain outside of the conference. The conference can not supply the storage and other equipment resources for use by the teams.

Size of the Storage System is Not the Basis for Success

The size of a storage system is not a basis for judging in the HPC Storage Challenge.As the HPC community matures there a large variance in the size of HPC storage systems is emerging.Often designs for large systems begin with smaller systems used as a proof of concept. Moreover, a great deal of creative solutions viewed as smaller can be as compelling as larger systems.Besides, size is a relative term and difficult to define without being arbitrary.�Things that determine the size of a storage system include any number of factors, not just the storage system itself (e.g., the number of nodes, number of processors per node, size of the network, amount of storage, bandwidth potential for the storage, etc.). Therefore, teams using systems of all sizes are encouraged to make submissions.

Finalists and Winners

Finalists will be notified by August 17, 2009 that they have been selected. Their sponsoring institutions may use this information for publicity. Each finalist team will be expected to make a presentation in a technical session during the conference. The schedule for the technical presentation sessions will be posted on the Storage Challenge web site several weeks after the notification date and listed in the final program.The Conference Program will also include a 150 word abstract and team member names along with their sponsoring institutions. The final version of this abstract and team list is due July 27, 2009 along with the project proposal. After the finalist teams have been notified, any changes made to the abstract and team list can not be reflected in the printed program. The winners will be announced in the conference awards session. Both finalists and winners will receive certificates for their accomplishments in this challenge.

Reports

Two principal reports must be submitted as part of the SC09 HPC Storage Challenge. The first one is the project proposal due July 27, 2009.The second one is the Final Project report due November 9, 2009.

Project Proposal

Participating teams are required to submit a Project Proposal by July 27, 2009 via the submission website which will be used by the judges to select finalists. Teams are encouraged to include results from proof of concept tests or works in progress to lend credibility to assertions made in their Project Proposal. A report in the form of a PDF file with at most 12 pages is to be included as part of the submission. A suggested format for this report follows

Title

Team Information

Abstract

Problem Statement (including hypothesis)

Hardware Configuration

Data and Storage Layout

Software Used (including application and systems software)

Description of the solution to the problem

Experiments to be done, and measurements and quantitative results to be collected

Include results from proof of concept tests or works in progress

Claims (how this project satisfies judging criteria)

Overall Performance

System and/or Application Scalability

Storage Resource Utilization

Innovation

Effectiveness

Acknowledgements

List of individuals or agencies not listed in the team information who made significant contributions to, or supported the work being reported

Final Project Report

Finalist teams in the SC09 HPC Storage Challenge are required to submit a Final Project Report by November 9, 2009 via the submission website which will be used by the judges to select the winner in each category. A report in the form of a PDF file with at most 12 pages is to be included as part of the submission. It must also include slides for the challenge finalist presentation in an appendix. Other appendices may also be included, but it is not guaranteed that they will be reviewed by the judges. The appendices are not included in the 12 page upper limit. A suggested format for this report follows

Title

Team Information

Abstract

Problem Statement (including hypothesis)

Hardware Configuration

Data and Storage Layout

Software Used (including application and systems software)

Description of the solution to the problem

Experiments, measurements and quantitative results of solution

Describe actual experiments and results

Describe how the actual measurements were made

Explain how they support the hypothesis

Describe conclusions based on the experiments

Describe how the judging criteria apply to the project

Overall Performance

System and/or Application Scalability

Storage Resource Utilization

Innovation

Effectiveness

Describe any deviations from the Project Proposal

Deviations from the Project Proposal as part of the exploratory process are anticipated, but the work must be in the same spirit as proposed

Price/performance information (see explanation below)

Appendices

Slides for challenge finalist presentation (required)

Other optional appendices

Acknowledgements: list of individuals or agencies not listed in the team information who made significant contributions to, or supported the work being reported

Finalist Interviews

Finalist teams in the SC09 HPC Storage Challenge are to participate in a private interview session with the Challenge committee. The purpose of this interview is to provide the committee with an opportunity to ask questions and seek clarifications that may arise from the finalist presentations and/or the Final Project Report.

The interviews are scheduled for Wednesday, November 18, 2009 starting at 8:00 AM.The exact interview schedule and location in the convention center will be sent to finalist teams via email.The schedule will also be posted on a sign just outside the interview room.

Judging

Project Proposal submissions and Final Project Report submissions to the SC09 HPC Storage Challenge will be evaluated by the committee based on a mixture of quantitative and qualitative criteria. Judging decisions are final.

Judging will be based on 3 broad categories: performance and scalability, effective use of storage and storage related resources, and innovation. The following list presents examples of things to be considered within each of these categories. These lists are neither exclusive nor exhaustive; it will be up to each team to select the appropriate measurements and analyses to make the case for their project.

Performance and Scalability

File size, application I/O request, file system block size

Application measurements (select appropriate measurement)

I/O rate in MB/s (both as an aggregate and harmonic mean)

IOP rate (both as an aggregate and harmonic mean)

File creations per second

Bandwidth per kilometer

System measurements

Bandwidth tools (e.g., iostat, netstat)

Latency tools (e.g., histograms of record size vs. time)

Storage system or vendor tools

Scalability; ability to expand solution to increasing numbers of

Disks and disk controllers

Nodes

Ensure that the magnitude of I/O operations is large enough to actually use disk

Effective use of resources

Measure storage and storage related resource utilization

disk/disk controllers, SAN, LAN

percent of peak BW per disk controller

percent of SAN/LAN utilization

assumes the application is storage I/O intensive

Requires "compelling" baseline measurements

Innovation

Novel ways to solve traditional and/or difficult problems

Effective use of new technologies

Well designed application

Best practices

The "Wow, that's COOL!" factor

Some Practical Advice

This is now the third year that the HPC Storage Challenge has been offered.These guidelines provide various suggested outlines and judging criteria. Experience shows that adhering to the outlines but more importantly providing complete information without being excessively wordy and explicitly addressing judging criteria is an effective strategy.