Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach

The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2015

Author dc:creator

• Luu, Huong Vu Thanh

Contributors dc:contributor

• Winslett, Marianne
• Gropp, William
• Snir, Marc
• Ross, Robert

• High Performance Computing (HPC)
• Input/Output (I/O)
• Parallel I/O
• Performance Analysis

Statement dc:rights

• Copyright 2015 Huong Vu Thanh Luu

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/78421

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/78421